Jul 23 15:46:36.862127 ip-10-0-128-121 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 23 15:46:36.862140 ip-10-0-128-121 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 23 15:46:36.862150 ip-10-0-128-121 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 23 15:46:36.862450 ip-10-0-128-121 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 23 15:46:45.901844 ip-10-0-128-121 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 23 15:46:45.901862 ip-10-0-128-121 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c2d6cb1393a64281b7cbb8a2302ef56c --
Jul 23 15:48:31.044389 ip-10-0-128-121 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.378774    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384445    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384456    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384460    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384463    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384466    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384468    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384471    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 15:48:31.640508 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384473    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 15:48:31.452941 ip-10-0-128-121 systemd[1]: Started Kubernetes Kubelet.
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384476    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384479    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384481    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384484    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384486    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384489    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384491    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384494    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384496    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384498    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384504    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384507    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384510    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384512    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384515    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384518    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384520    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384523    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384526    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 15:48:31.655777 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384528    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384530    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384533    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384536    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384539    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384541    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384544    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384546    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384549    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384551    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384554    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384556    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384559    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384561    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384564    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384566    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384568    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384571    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384573    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384575    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 15:48:31.657219 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384579    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384581    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384584    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384587    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384590    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384592    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384594    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384597    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384599    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384602    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384604    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384607    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384609    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384613    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384617    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384621    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384624    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384627    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384630    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384633    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 15:48:31.658241 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384635    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384638    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384640    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384643    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384646    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384648    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384652    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384654    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384656    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384659    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384661    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384664    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384666    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384669    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384672    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384674    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384677    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384679    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384682    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384684    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 15:48:31.658819 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384687    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384689    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384692    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384694    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384696    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384698    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384701    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384704    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384708    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384713    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384716    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.384719    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385400    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385407    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385410    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385413    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385416    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385419    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385421    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 15:48:31.659457 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385424    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385427    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385431    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385434    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385437    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385439    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385442    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385444    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385448    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385451    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385453    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385456    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385458    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385461    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385464    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385467    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385469    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385471    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385474    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 15:48:31.659994 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385476    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385479    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385482    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385484    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385487    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385490    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385492    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385495    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385497    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385500    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385503    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385506    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385508    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385511    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385513    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385517    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385520    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385524    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385527    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385532    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 15:48:31.660634 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385535    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385537    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385540    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385543    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385545    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385548    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385551    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385553    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385559    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385562    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385565    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385567    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385570    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385572    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385575    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385577    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385580    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385583    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385585    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 15:48:31.661209 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385588    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385590    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385593    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385595    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385598    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385600    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385602    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385605    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385607    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385610    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385613    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385615    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385618    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385620    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385623    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385625    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385628    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385631    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385633    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385636    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 15:48:31.661778 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385640    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385647    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385650    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385653    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385655    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385658    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385660    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385663    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385665    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385668    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385670    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385672    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385675    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.385677    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386040    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386045    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386049    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386051    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386055    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386057    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 15:48:31.662399 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386060    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386062    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386065    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386067    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386070    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386072    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386075    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386077    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386090    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386094    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386097    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386100    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386104    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386111    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386115    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386151    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386155    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386159    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386163    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386167    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 15:48:31.663028 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386171    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386175    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386178    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386182    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386185    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386189    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386193    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386197    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386200    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386204    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386208    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386211    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386215    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386218    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386222    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386226    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386230    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386233    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386237    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386241    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 15:48:31.663627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386245    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386249    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386253    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386257    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386260    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386267    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386270    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386274    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386278    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386282    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386285    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386290    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386294    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386298    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386304    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386309    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386313    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386318    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386321    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386325    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 15:48:31.664183 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386329    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386333    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386337    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386342    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386347    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386353    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386356    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386360    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386364    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386369    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386375    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386378    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386383    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386387    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386391    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386395    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386399    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386407    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386411    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 15:48:31.664769 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386414    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386418    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386421    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386425    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386429    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386432    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386436    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386440    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386444    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386448    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386452    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386455    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386459    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.386463    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386554    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386563    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386573    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386578    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386584    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386588    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386595    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 15:48:31.665322 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386600    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386605    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386609    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386614    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386618    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386623    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386627    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386632    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386636    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386643    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386647    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386653    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386657    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386662    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386667    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386671    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386677    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386681    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386685    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386701    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386721    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386739    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386744    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386748    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 15:48:31.665838 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386753    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386765    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386769    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386772    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386775    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386778    2555 flags.go:64] FLAG: --enable-server="true"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386781    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386789    2555 flags.go:64] FLAG: --event-burst="100"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386793    2555 flags.go:64] FLAG: --event-qps="50"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386796    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386801    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386804    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386809    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386812    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386815    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386818    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386821    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386824    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386827    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386830    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386835    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386838    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386841    2555 flags.go:64] FLAG: --feature-gates=""
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386845    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386848    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 15:48:31.666534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386851    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386854    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386857    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386861    2555 flags.go:64] FLAG: --help="false"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386864    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-128-121.ec2.internal"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386867    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386870    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386873    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386876    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386881    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386884    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386886    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386889    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386892    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386895    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386898    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386901    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386904    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386917    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386922    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386925    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386928    2555 flags.go:64] FLAG: --lock-file=""
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386931    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386934    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 15:48:31.667241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386937    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386942    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386945    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386949    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386952    2555 flags.go:64] FLAG: --logging-format="text"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386956    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386959    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386962    2555 flags.go:64] FLAG: --manifest-url=""
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386965    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386969    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386972    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386977    2555 flags.go:64] FLAG: --max-pods="110"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386980    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386983    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386986    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386989    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386992    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386995    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.386998    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387006    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387008    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387011    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387014    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 23 15:48:31.667894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387018    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387023    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387025    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387028    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387032    2555 flags.go:64] FLAG: --port="10250"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387035    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387038    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03b83cf8a299138c0"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387041    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387044    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387047    2555 flags.go:64] FLAG: --register-node="true"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387050    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387054    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387057    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387061    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387064    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387068    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387070    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387073    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387076    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387079    2555 flags.go:64] FLAG: --runonce="false"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387082    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387085    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387088    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387092    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387095    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387098    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 15:48:31.668768 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387101    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387104    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387107    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387110    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387113    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387115    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387118    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387121    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387124    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387149    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387153    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387157    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387161    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387164    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387167    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387170    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387173    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387176    2555 flags.go:64] FLAG: --v="2"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387180    2555 flags.go:64] FLAG: --version="false"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387186    2555 flags.go:64] FLAG: --vmodule=""
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387190    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.387193    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387327    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387331    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 15:48:31.669483 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387335    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387338    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387341    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387344    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387347    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387351    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387354    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387358    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387362    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387366    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387368    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387371    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387374    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387376    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387379    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387381    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387384    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387388    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387391    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 15:48:31.842377 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387394    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387400    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387403    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387406    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387409    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387413    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387415    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387418    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387422    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387425    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387428    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387431    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387433    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387436    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387439    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387441    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387444    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387446    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387450    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387453    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 15:48:31.843006 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387455    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387458    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387460    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387463    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387466    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387468    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387470    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387473    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387476    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387478    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387481    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387495    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387499    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387502    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387505    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387508    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387510    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387513    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387515    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387518    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 15:48:31.843624 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387521    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387524    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387526    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387529    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387531    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387534    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387536    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387539    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387542    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387544    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387548    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387551    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387554    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387558    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387560    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387563    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387566    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387568    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387570    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387573    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 15:48:31.844192 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387576    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387578    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387581    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387583    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387586    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387588    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387590    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387594    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387597    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387600    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387602    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387605    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387608    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387611    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387613    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387616    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387618    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 15:48:31.844699 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.387621    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.388165    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.395720    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.395739    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.395758    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.395764    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395818    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395823    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395827    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395830    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395833    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395835    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395839    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395841    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395844    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 15:48:31.845187 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395846    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395849    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395852    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395854    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395857    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395860    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395862    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395866    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395868    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395871    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395874    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395876    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395879    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395882    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395884    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395887    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395889    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395892    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395895    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395898    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 15:48:31.845592 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395901    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395904    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395909    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395912    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395914    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395917    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395920    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395922    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395925    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395928    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395930    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395933    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395935    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395938    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395940    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395943    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395947    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395951    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395953    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395956    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 15:48:31.846182 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395959    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395962    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395964    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395969    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395971    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395974    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395976    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395979    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395981    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395984    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395986    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395989    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395991    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395994    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.395998    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396002    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396005    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396007    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396010    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 15:48:31.846698 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396012    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396015    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396018    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396020    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396023    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396025    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396028    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396030    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396033    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396035    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396038    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396040    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396044    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396048    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396050    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396053    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396057    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396060    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396063    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396066    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 15:48:31.847246 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396068    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396071    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396074    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396077    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396079    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396081    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396084    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396088    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396091    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396093    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396096    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.396100    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396239    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396245    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396248    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 15:48:31.847760 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396251    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396254    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396257    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396260    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396262    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396265    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396267    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396270    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396272    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396276    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396278    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396281    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396283    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396287    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396290    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396292    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396295    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396298    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396300    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396303    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 15:48:31.848222 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396305    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396308    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396310    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396313    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396315    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396319    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396322    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396324    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396327    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396329    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396333    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396336    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396339    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396342    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396344    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396347    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396349    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396352    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396354    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396357    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 15:48:31.848729 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396359    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396362    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396365    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396367    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396370    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396373    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396376    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396378    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396381    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396383    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396386    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396388    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396390    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396393    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396396    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396398    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396400    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396404    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396407    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396409    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 15:48:31.849295 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396412    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396415    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396419    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396421    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396424    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396427    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396429    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396432    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396434    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396437    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396439    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396442    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396444    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396446    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396449    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396452    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396455    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396458    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396461    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 15:48:31.849801 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396464    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396466    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396469    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396471    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396474    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396476    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396479    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396481    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396484    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396486    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396490    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396492    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396495    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396498    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396501    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396504    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 15:48:31.850414 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:31.396507    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.396511    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.396607    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.399205    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.400485    2555 server.go:1019] "Starting client certificate rotation"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.400580    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.400617    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.403498    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.407762    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.421376    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.423513    2555 log.go:25] "Validated CRI v1 image API"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.424371    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 15:48:31.850845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.426619    2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d10ed2e1-6167-4ed9-8fd0-a513f1bd1142:/dev/nvme0n1p4 e50f6146-cb32-4ea4-890b-72e76e2d7b88:/dev/nvme0n1p3]
Jul 23 15:48:31.851275 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.426633    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 15:48:31.851275 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.428880    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.433203    2555 manager.go:216] Machine: {Timestamp:2026-07-23 15:48:31.431409562 +0000 UTC m=+0.281564508 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3192061 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22d5cac61c5e3492ed4cd462d36b84 SystemUUID:ec22d5ca-c61c-5e34-92ed-4cd462d36b84 BootID:c2d6cb13-93a6-4281-b7cb-b8a2302ef56c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d5:e0:27:be:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d5:e0:27:be:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:82:39:10:a6:0c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.433297    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.433380    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.436042    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.436085    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.436903    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.436913    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.436921    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.437544    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.438929    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.442914    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.443023    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.443049    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.443058    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.443072    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.443080    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.443088    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 15:48:31.851456 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.443559    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.443581    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.445623    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.447473    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.447499    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448187    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448202    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448208    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448213    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448220    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448227    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448235    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448240    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448246    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448251    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448260    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448265    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448849    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.448856    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.452311    2555 server.go:1268] "Started kubelet"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.452968    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.453045    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.453155    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.453191    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.453223    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.453410    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.454305    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.455245    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.457725    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.459499    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2jfkg"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461232    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461251    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.461598    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461865    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.460811    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-121.ec2.internal.18c4f5e3e284c3db  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-121.ec2.internal,UID:ip-10-0-128-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-121.ec2.internal,},FirstTimestamp:2026-07-23 15:48:31.452292059 +0000 UTC m=+0.302447004,LastTimestamp:2026-07-23 15:48:31.452292059 +0000 UTC m=+0.302447004,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-121.ec2.internal,}"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461866    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461889    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461944    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461954    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461985    2555 factory.go:55] Registering systemd factory
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.461998    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 23 15:48:32.034373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.462282    2555 factory.go:153] Registering CRI-O factory
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.462294    2555 factory.go:223] Registration of the crio container factory successfully
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.462335    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.462357    2555 factory.go:103] Registering Raw factory
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.462370    2555 manager.go:1196] Started watching for new ooms in manager
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.462728    2555 manager.go:349] Starting recovery of all containers
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.465184    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.465318    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.468075    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2jfkg"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.476172    2555 manager.go:354] Recovery completed
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.480762    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.482849    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.482877    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.482890    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.483465    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.483473    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.483487    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.483472    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.484246    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.484267    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.484276    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.484938    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-121.ec2.internal.18c4f5e3e4573c9f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-121.ec2.internal,UID:ip-10-0-128-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-121.ec2.internal,},FirstTimestamp:2026-07-23 15:48:31.482862751 +0000 UTC m=+0.333017700,LastTimestamp:2026-07-23 15:48:31.482862751 +0000 UTC m=+0.333017700,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-121.ec2.internal,}"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.485505    2555 policy_none.go:49] "None policy: Start"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.485517    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.485527    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.487503    2555 policy_none.go:47] "Start"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.518952    2555 manager.go:337] "Starting Device Plugin manager"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.518984    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.518996    2555 server.go:87] "Starting device plugin registration server"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.519237    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.519249    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.519350    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.519455    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.519466    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.520296    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.520336    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.577586    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.578690    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.578721    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.578739    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 15:48:32.035673 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.578767    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.581300    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.619451    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.620208    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.620233    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.620243    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.620265    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.628251    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.628267    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-121.ec2.internal\": node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.642211    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.678937    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"]
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.679005    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.680493    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.680522    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.680531    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.682733    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.682839    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.682867    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.683438    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.683457    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.683469    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.683509    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.683525    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.683540    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.685822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.685853    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.687221    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.687244    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.687255    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeHasSufficientPID"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.725563    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-121.ec2.internal\" not found" node="ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.735451    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-121.ec2.internal\" not found" node="ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.742444    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.763568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c061e960ca96a1ef3107eae942918a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"6c061e960ca96a1ef3107eae942918a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.037447 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.763596    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c061e960ca96a1ef3107eae942918a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"6c061e960ca96a1ef3107eae942918a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.763627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0b8168cb74fd246a62dfc8f2491ac00-config\") pod \"kube-apiserver-proxy-ip-10-0-128-121.ec2.internal\" (UID: \"d0b8168cb74fd246a62dfc8f2491ac00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.842868    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.864203    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c061e960ca96a1ef3107eae942918a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"6c061e960ca96a1ef3107eae942918a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.864227    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c061e960ca96a1ef3107eae942918a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"6c061e960ca96a1ef3107eae942918a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.864250    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0b8168cb74fd246a62dfc8f2491ac00-config\") pod \"kube-apiserver-proxy-ip-10-0-128-121.ec2.internal\" (UID: \"d0b8168cb74fd246a62dfc8f2491ac00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.864294    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c061e960ca96a1ef3107eae942918a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"6c061e960ca96a1ef3107eae942918a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.864275    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d0b8168cb74fd246a62dfc8f2491ac00-config\") pod \"kube-apiserver-proxy-ip-10-0-128-121.ec2.internal\" (UID: \"d0b8168cb74fd246a62dfc8f2491ac00\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:31.864327    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c061e960ca96a1ef3107eae942918a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal\" (UID: \"6c061e960ca96a1ef3107eae942918a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:31.943453    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:32.038384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:32.028084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:33.434509 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:32.038958    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:33.434509 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:32.043806    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:33.434509 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:33.369253    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:33.588792 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:33.469798    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:33.588792 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:33.570350    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:33.756775 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:33.599468    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 15:48:33.756775 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:33.624920    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 15:48:33.756775 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:33.625025    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 15:48:33.756775 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:33.625026    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 15:48:33.756775 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:33.671430    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:33.756775 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:33.695090    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 15:48:33.756775 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:33.702151    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 15:43:32 +0000 UTC" deadline="2028-03-13 01:56:43.061781487 +0000 UTC"
Jul 23 15:48:33.756775 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:33.702173    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14362h8m9.35961144s"
Jul 23 15:48:33.898838 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:33.772186    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:33.898838 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:33.872287    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:34.027460 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:33.972780    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-121.ec2.internal\" not found"
Jul 23 15:48:34.027460 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:34.021464    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b8168cb74fd246a62dfc8f2491ac00.slice/crio-a881a94f4543d95180eb5bc8a5d713e1af1e7c4dd398d9acdc2bf9d8028c24d8 WatchSource:0}: Error finding container a881a94f4543d95180eb5bc8a5d713e1af1e7c4dd398d9acdc2bf9d8028c24d8: Status 404 returned error can't find the container with id a881a94f4543d95180eb5bc8a5d713e1af1e7c4dd398d9acdc2bf9d8028c24d8
Jul 23 15:48:34.027460 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.027008    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 15:48:34.202695 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.060431    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 15:48:34.202695 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.087267    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:34.202695 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.097163    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 15:48:34.202695 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.098127    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal"
Jul 23 15:48:34.202695 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.106596    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.668558    2555 apiserver.go:52] "Watching apiserver"
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.675778    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.677737    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-fz59x","openshift-multus/network-metrics-daemon-rghs4","openshift-network-diagnostics/network-check-target-zr2d7","openshift-network-operator/iptables-alerter-vvcgh","kube-system/global-pull-secret-syncer-glxtl","kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf","openshift-image-registry/node-ca-t82bm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal","openshift-multus/multus-sq4qw","openshift-ovn-kubernetes/ovnkube-node-29d6g","kube-system/konnectivity-agent-skg42","openshift-cluster-node-tuning-operator/tuned-pnnqq","openshift-dns/node-resolver-gdwzk"]
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.680256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.682948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.685023    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.685096    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.685342    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.685369    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 15:48:34.685514 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.685519    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 15:48:34.686434 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.685660    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hsldd\""
Jul 23 15:48:34.686434 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.686390    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kjwcc\""
Jul 23 15:48:34.686730 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.686469    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 15:48:34.688085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.686835    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 15:48:34.688085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.687001    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 15:48:34.688085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.687373    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.689984 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.689773    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:34.689984 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.689837    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:34.694536 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.692822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.695818    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx57s\" (UniqueName: \"kubernetes.io/projected/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-kube-api-access-nx57s\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.695854    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.695882    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-kubernetes\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.695912    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysctl-conf\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.695935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-tmp\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.695958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfcrj\" (UniqueName: \"kubernetes.io/projected/f3caef45-ca17-4dd6-b51a-18b82c00b863-kube-api-access-zfcrj\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.695982    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.695989    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696005    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-socket-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696028    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysconfig\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696051    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-tuned\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696074    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3caef45-ca17-4dd6-b51a-18b82c00b863-iptables-alerter-script\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696096    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3caef45-ca17-4dd6-b51a-18b82c00b863-host-slash\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-registration-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-device-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.697723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-etc-selinux\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdll2\" (UniqueName: \"kubernetes.io/projected/f05ff6b4-6863-4fdb-974e-9be9807c132a-kube-api-access-cdll2\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-modprobe-d\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysctl-d\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-lib-modules\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696295    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-var-lib-kubelet\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696366    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-sys-fs\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696409    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-systemd\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696441    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-run\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696461    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-sys\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.696476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-host\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.698428    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.699218    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.699324    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8mgcn\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.699366    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.700425    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.700953    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.701058    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.701346    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.701420    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.702001    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.702332    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.702474    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.702657    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 15:48:34.782506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.702778    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.702840    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ls6rf\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.703246    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.703481    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zqw9x\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.703635    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.703874    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.707972    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9ps7f\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.708545    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.709258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.711691    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.711837    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ck25d\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.712458    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.714674    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vn86t\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.715125    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.715191    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.715359    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.715404    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.715942    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.715971    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.716259    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.718006    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.718034    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.718106    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v4qlf\""
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.730203    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wwjbb"
Jul 23 15:48:34.783547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.730290    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wwjbb"
Jul 23 15:48:34.788021 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.787823    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-device-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-etc-selinux\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797294    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-lib-modules\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797312    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx57s\" (UniqueName: \"kubernetes.io/projected/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-kube-api-access-nx57s\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-device-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797338    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovnkube-script-lib\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9-konnectivity-ca\") pod \"konnectivity-agent-skg42\" (UID: \"fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9\") " pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797402    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-etc-selinux\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-sys-fs\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797451    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-lib-modules\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797464    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-systemd\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797506    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-sys-fs\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797512    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-systemd\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797534    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-cni-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797560    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-os-release\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797581    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-netns\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.885105 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797615    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-kubelet\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797663    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-ovn\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797797    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-os-release\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-tmp\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797851    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysconfig\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797898    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-tuned\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797920    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3caef45-ca17-4dd6-b51a-18b82c00b863-host-slash\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797942    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-cnibin\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797966    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-cni-bin\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.797994    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798040    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9-agent-certs\") pod \"konnectivity-agent-skg42\" (UID: \"fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9\") " pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798062    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51b14893-b696-45a2-8d6e-78cf129dd263-multus-daemon-config\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798085    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-run-netns\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-run-ovn-kubernetes\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798147    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dffd5fde-6708-475a-ae48-e3f9b983009a-dbus\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:34.888022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798169    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-system-cni-dir\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798205    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8444dfcc-a56d-4220-86eb-2f7b56291263-hosts-file\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-conf-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-etc-kubernetes\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798338    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798364    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtmz\" (UniqueName: \"kubernetes.io/projected/8444dfcc-a56d-4220-86eb-2f7b56291263-kube-api-access-qxtmz\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798419    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-socket-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798446    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-kubelet\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798484    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls2ph\" (UniqueName: \"kubernetes.io/projected/51b14893-b696-45a2-8d6e-78cf129dd263-kube-api-access-ls2ph\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-systemd-units\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798570    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-env-overrides\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798597    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs55r\" (UniqueName: \"kubernetes.io/projected/e01e18b9-f07e-494f-9a2d-50f2f70471de-kube-api-access-vs55r\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798628    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdll2\" (UniqueName: \"kubernetes.io/projected/f05ff6b4-6863-4fdb-974e-9be9807c132a-kube-api-access-cdll2\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798652    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51b14893-b696-45a2-8d6e-78cf129dd263-cni-binary-copy\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.888777 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798675    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-k8s-cni-cncf-io\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798740    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-systemd\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798799    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-log-socket\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798821    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bddfe96-b526-4de7-981b-de1f84b81b3c-serviceca\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52xt5\" (UniqueName: \"kubernetes.io/projected/1bddfe96-b526-4de7-981b-de1f84b81b3c-kube-api-access-52xt5\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798884    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798907    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-slash\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798938    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovnkube-config\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-cni-multus\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.798983    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovn-node-metrics-cert\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8444dfcc-a56d-4220-86eb-2f7b56291263-tmp-dir\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799055    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-system-cni-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cnibin\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799104    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znkkk\" (UniqueName: \"kubernetes.io/projected/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-kube-api-access-znkkk\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-registration-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.889973 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-modprobe-d\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799240    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysctl-d\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799263    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-var-lib-kubelet\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-host\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799309    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-socket-dir-parent\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799334    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-run\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799356    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-sys\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-hostroot\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-var-lib-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799425    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-cni-bin\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799457    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-cni-netd\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dffd5fde-6708-475a-ae48-e3f9b983009a-kubelet-config\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799509    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-kubernetes\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799539    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysctl-conf\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799563    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfcrj\" (UniqueName: \"kubernetes.io/projected/f3caef45-ca17-4dd6-b51a-18b82c00b863-kube-api-access-zfcrj\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799588    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-node-log\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799624    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.890863 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbk6\" (UniqueName: \"kubernetes.io/projected/46c729a0-b515-4581-9a67-1ee16d3649ab-kube-api-access-jpbk6\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799709    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bddfe96-b526-4de7-981b-de1f84b81b3c-host\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799735    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3caef45-ca17-4dd6-b51a-18b82c00b863-iptables-alerter-script\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799792    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-multus-certs\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799819    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-etc-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.799844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800239    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800310    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800366    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3caef45-ca17-4dd6-b51a-18b82c00b863-host-slash\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800429    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-socket-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800612    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysctl-d\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800732    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-kubernetes\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800842    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-modprobe-d\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800877    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-run\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-host\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.801005    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-var-lib-kubelet\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.801356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-sys\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.800255    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysconfig\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.891766 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.801387    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-sysctl-conf\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.801571    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f05ff6b4-6863-4fdb-974e-9be9807c132a-registration-dir\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.802480    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3caef45-ca17-4dd6-b51a-18b82c00b863-iptables-alerter-script\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.803949    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-etc-tuned\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.805934    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-tmp\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.808629    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" event={"ID":"d0b8168cb74fd246a62dfc8f2491ac00","Type":"ContainerStarted","Data":"a881a94f4543d95180eb5bc8a5d713e1af1e7c4dd398d9acdc2bf9d8028c24d8"}
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.809799    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.809829    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.809865    2555 projected.go:196] Error preparing data for projected volume kube-api-access-q7mx8 for pod openshift-network-diagnostics/network-check-target-zr2d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.809948    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8 podName:d443f735-cecc-4b70-95e8-3bce5625a11c nodeName:}" failed. No retries permitted until 2026-07-23 15:48:35.309927972 +0000 UTC m=+2.935462447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q7mx8" (UniqueName: "kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8") pod "network-check-target-zr2d7" (UID: "d443f735-cecc-4b70-95e8-3bce5625a11c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.813028    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx57s\" (UniqueName: \"kubernetes.io/projected/d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7-kube-api-access-nx57s\") pod \"tuned-pnnqq\" (UID: \"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7\") " pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.813056    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfcrj\" (UniqueName: \"kubernetes.io/projected/f3caef45-ca17-4dd6-b51a-18b82c00b863-kube-api-access-zfcrj\") pod \"iptables-alerter-vvcgh\" (UID: \"f3caef45-ca17-4dd6-b51a-18b82c00b863\") " pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:34.893058 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.813768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdll2\" (UniqueName: \"kubernetes.io/projected/f05ff6b4-6863-4fdb-974e-9be9807c132a-kube-api-access-cdll2\") pod \"aws-ebs-csi-driver-node-cr2kf\" (UID: \"f05ff6b4-6863-4fdb-974e-9be9807c132a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.901997    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-slash\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovnkube-config\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-cni-multus\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovn-node-metrics-cert\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902121    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-slash\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902134    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-cni-multus\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902169    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8444dfcc-a56d-4220-86eb-2f7b56291263-tmp-dir\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902195    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-system-cni-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902216    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cnibin\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902237    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znkkk\" (UniqueName: \"kubernetes.io/projected/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-kube-api-access-znkkk\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902271    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-socket-dir-parent\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902296    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-hostroot\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902311    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-var-lib-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902324    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-cni-bin\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902338    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-cni-netd\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.987485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dffd5fde-6708-475a-ae48-e3f9b983009a-kubelet-config\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902377    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-node-log\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902400    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902424    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbk6\" (UniqueName: \"kubernetes.io/projected/46c729a0-b515-4581-9a67-1ee16d3649ab-kube-api-access-jpbk6\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-socket-dir-parent\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bddfe96-b526-4de7-981b-de1f84b81b3c-host\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902473    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-multus-certs\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902478    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8444dfcc-a56d-4220-86eb-2f7b56291263-tmp-dir\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-system-cni-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-etc-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902521    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902524    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cnibin\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovnkube-script-lib\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9-konnectivity-ca\") pod \"konnectivity-agent-skg42\" (UID: \"fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9\") " pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-cni-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902649    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-os-release\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902714    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-netns\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902765    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-kubelet\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.988877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902790    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-ovn\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902825    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-os-release\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902883    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-cnibin\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-cni-bin\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902949    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-hostroot\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902967    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9-agent-certs\") pod \"konnectivity-agent-skg42\" (UID: \"fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9\") " pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903032    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51b14893-b696-45a2-8d6e-78cf129dd263-multus-daemon-config\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.903042    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903035    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dffd5fde-6708-475a-ae48-e3f9b983009a-kubelet-config\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.903100    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret podName:dffd5fde-6708-475a-ae48-e3f9b983009a nodeName:}" failed. No retries permitted until 2026-07-23 15:48:35.403082745 +0000 UTC m=+3.028617233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret") pod "global-pull-secret-syncer-glxtl" (UID: "dffd5fde-6708-475a-ae48-e3f9b983009a") : object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903103    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-node-log\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903153    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-run-netns\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903235    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-cni-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:34.989463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovnkube-script-lib\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903297    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-os-release\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903311    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bddfe96-b526-4de7-981b-de1f84b81b3c-host\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.902649    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovnkube-config\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903355    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-multus-certs\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903359    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-netns\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903398    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-etc-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-ovn\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903434    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-kubelet\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903467    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-os-release\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.903537    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:34.903581    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs podName:46c729a0-b515-4581-9a67-1ee16d3649ab nodeName:}" failed. No retries permitted until 2026-07-23 15:48:35.403564143 +0000 UTC m=+3.029098610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs") pod "network-metrics-daemon-rghs4" (UID: "46c729a0-b515-4581-9a67-1ee16d3649ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903638    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-cnibin\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903681    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-cni-bin\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903723    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903776    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-var-lib-openvswitch\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903809    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-cni-bin\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.132900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903868    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9-konnectivity-ca\") pod \"konnectivity-agent-skg42\" (UID: \"fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9\") " pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-run-netns\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903916    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-cni-netd\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-run-ovn-kubernetes\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dffd5fde-6708-475a-ae48-e3f9b983009a-dbus\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903923    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.903987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-system-cni-dir\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904017    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-run-ovn-kubernetes\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904033    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904040    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dffd5fde-6708-475a-ae48-e3f9b983009a-dbus\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904059    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8444dfcc-a56d-4220-86eb-2f7b56291263-hosts-file\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904072    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-system-cni-dir\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-conf-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-etc-kubernetes\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904128    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8444dfcc-a56d-4220-86eb-2f7b56291263-hosts-file\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtmz\" (UniqueName: \"kubernetes.io/projected/8444dfcc-a56d-4220-86eb-2f7b56291263-kube-api-access-qxtmz\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:35.135119 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-multus-conf-dir\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904190    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-etc-kubernetes\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904200    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-kubelet\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls2ph\" (UniqueName: \"kubernetes.io/projected/51b14893-b696-45a2-8d6e-78cf129dd263-kube-api-access-ls2ph\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904262    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-systemd-units\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-env-overrides\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904307    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51b14893-b696-45a2-8d6e-78cf129dd263-multus-daemon-config\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904317    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-var-lib-kubelet\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904312    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs55r\" (UniqueName: \"kubernetes.io/projected/e01e18b9-f07e-494f-9a2d-50f2f70471de-kube-api-access-vs55r\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51b14893-b696-45a2-8d6e-78cf129dd263-cni-binary-copy\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904388    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-systemd-units\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-k8s-cni-cncf-io\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904428    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51b14893-b696-45a2-8d6e-78cf129dd263-host-run-k8s-cni-cncf-io\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904433    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-systemd\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904468    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-run-systemd\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904491    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-log-socket\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.135758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904524    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bddfe96-b526-4de7-981b-de1f84b81b3c-serviceca\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904563    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e01e18b9-f07e-494f-9a2d-50f2f70471de-log-socket\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904577    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e01e18b9-f07e-494f-9a2d-50f2f70471de-ovn-node-metrics-cert\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904592    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52xt5\" (UniqueName: \"kubernetes.io/projected/1bddfe96-b526-4de7-981b-de1f84b81b3c-kube-api-access-52xt5\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904851    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e01e18b9-f07e-494f-9a2d-50f2f70471de-env-overrides\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904948    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bddfe96-b526-4de7-981b-de1f84b81b3c-serviceca\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.904957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51b14893-b696-45a2-8d6e-78cf129dd263-cni-binary-copy\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.905255    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9-agent-certs\") pod \"konnectivity-agent-skg42\" (UID: \"fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9\") " pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.911911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znkkk\" (UniqueName: \"kubernetes.io/projected/2dae43a1-ca00-4cf2-b8c0-ba84573a868a-kube-api-access-znkkk\") pod \"multus-additional-cni-plugins-fz59x\" (UID: \"2dae43a1-ca00-4cf2-b8c0-ba84573a868a\") " pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.920597    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls2ph\" (UniqueName: \"kubernetes.io/projected/51b14893-b696-45a2-8d6e-78cf129dd263-kube-api-access-ls2ph\") pod \"multus-sq4qw\" (UID: \"51b14893-b696-45a2-8d6e-78cf129dd263\") " pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.920965    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52xt5\" (UniqueName: \"kubernetes.io/projected/1bddfe96-b526-4de7-981b-de1f84b81b3c-kube-api-access-52xt5\") pod \"node-ca-t82bm\" (UID: \"1bddfe96-b526-4de7-981b-de1f84b81b3c\") " pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.921116    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbk6\" (UniqueName: \"kubernetes.io/projected/46c729a0-b515-4581-9a67-1ee16d3649ab-kube-api-access-jpbk6\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.921188    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtmz\" (UniqueName: \"kubernetes.io/projected/8444dfcc-a56d-4220-86eb-2f7b56291263-kube-api-access-qxtmz\") pod \"node-resolver-gdwzk\" (UID: \"8444dfcc-a56d-4220-86eb-2f7b56291263\") " pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.922089    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs55r\" (UniqueName: \"kubernetes.io/projected/e01e18b9-f07e-494f-9a2d-50f2f70471de-kube-api-access-vs55r\") pod \"ovnkube-node-29d6g\" (UID: \"e01e18b9-f07e-494f-9a2d-50f2f70471de\") " pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:34.995672    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.003342    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pnnqq"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.005368    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:35.006050    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05ff6b4_6863_4fdb_974e_9be9807c132a.slice/crio-8955a4d19f47811770e269b7c7920c9774be5a2f43b577910c03a2cea8f36247 WatchSource:0}: Error finding container 8955a4d19f47811770e269b7c7920c9774be5a2f43b577910c03a2cea8f36247: Status 404 returned error can't find the container with id 8955a4d19f47811770e269b7c7920c9774be5a2f43b577910c03a2cea8f36247
Jul 23 15:48:35.136501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.014366    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vvcgh"
Jul 23 15:48:35.137220 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:35.017113    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96f9a1a_32c4_4c5c_bd5c_0c039d192cb7.slice/crio-10ba7015a0bb19db55b592e095fe778e560af044443ad16eb5c8da06ff9d438a WatchSource:0}: Error finding container 10ba7015a0bb19db55b592e095fe778e560af044443ad16eb5c8da06ff9d438a: Status 404 returned error can't find the container with id 10ba7015a0bb19db55b592e095fe778e560af044443ad16eb5c8da06ff9d438a
Jul 23 15:48:35.137220 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.020149    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gdwzk"
Jul 23 15:48:35.137220 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:35.026074    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3caef45_ca17_4dd6_b51a_18b82c00b863.slice/crio-8fd74568038a0e5a8672aa67f5b7388da35a6b44b234da95d0e973cd9b1c3153 WatchSource:0}: Error finding container 8fd74568038a0e5a8672aa67f5b7388da35a6b44b234da95d0e973cd9b1c3153: Status 404 returned error can't find the container with id 8fd74568038a0e5a8672aa67f5b7388da35a6b44b234da95d0e973cd9b1c3153
Jul 23 15:48:35.137220 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.027807    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t82bm"
Jul 23 15:48:35.137220 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.036422    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fz59x"
Jul 23 15:48:35.137220 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.043293    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sq4qw"
Jul 23 15:48:35.137220 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.050870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:48:35.137220 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.055423    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:35.379169 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:35.379137    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8444dfcc_a56d_4220_86eb_2f7b56291263.slice/crio-398cf6ce56380684bbf2ff11b5437ab509370145e60a2cc558b111f6ac9e2eb6 WatchSource:0}: Error finding container 398cf6ce56380684bbf2ff11b5437ab509370145e60a2cc558b111f6ac9e2eb6: Status 404 returned error can't find the container with id 398cf6ce56380684bbf2ff11b5437ab509370145e60a2cc558b111f6ac9e2eb6
Jul 23 15:48:35.407731 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.407669    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:35.407731 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.407710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:35.407913 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.407828    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:35.407913 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.407843    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:35.407913 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.407854    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:35.407913 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.407885    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret podName:dffd5fde-6708-475a-ae48-e3f9b983009a nodeName:}" failed. No retries permitted until 2026-07-23 15:48:36.407866032 +0000 UTC m=+4.033400487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret") pod "global-pull-secret-syncer-glxtl" (UID: "dffd5fde-6708-475a-ae48-e3f9b983009a") : object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:35.407913 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.407907    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs podName:46c729a0-b515-4581-9a67-1ee16d3649ab nodeName:}" failed. No retries permitted until 2026-07-23 15:48:36.407897088 +0000 UTC m=+4.033431541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs") pod "network-metrics-daemon-rghs4" (UID: "46c729a0-b515-4581-9a67-1ee16d3649ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:35.408172 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.407944    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 15:48:35.408172 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.407959    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 15:48:35.408172 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.407972    2555 projected.go:196] Error preparing data for projected volume kube-api-access-q7mx8 for pod openshift-network-diagnostics/network-check-target-zr2d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:35.408172 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.408013    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8 podName:d443f735-cecc-4b70-95e8-3bce5625a11c nodeName:}" failed. No retries permitted until 2026-07-23 15:48:36.408002104 +0000 UTC m=+4.033536584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7mx8" (UniqueName: "kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8") pod "network-check-target-zr2d7" (UID: "d443f735-cecc-4b70-95e8-3bce5625a11c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:35.719787 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.719558    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 15:48:35.733882 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.733834    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 15:43:34 +0000 UTC" deadline="2028-01-11 14:10:07.006133419 +0000 UTC"
Jul 23 15:48:35.733882 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.733875    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12886h21m31.272262867s"
Jul 23 15:48:35.805908 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.804116    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:35.805908 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:35.804469    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b14893_b696_45a2_8d6e_78cf129dd263.slice/crio-d3a64e3906ead25e157ba7af3084f9342aa790f6dea7965471a8254ef5420043 WatchSource:0}: Error finding container d3a64e3906ead25e157ba7af3084f9342aa790f6dea7965471a8254ef5420043: Status 404 returned error can't find the container with id d3a64e3906ead25e157ba7af3084f9342aa790f6dea7965471a8254ef5420043
Jul 23 15:48:35.805908 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:35.804838    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:35.805908 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:35.805618    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bddfe96_b526_4de7_981b_de1f84b81b3c.slice/crio-d46321ef5dfb00a9102e10e88b8d4a3772a632b14e7d313fa231233a3f1dbb25 WatchSource:0}: Error finding container d46321ef5dfb00a9102e10e88b8d4a3772a632b14e7d313fa231233a3f1dbb25: Status 404 returned error can't find the container with id d46321ef5dfb00a9102e10e88b8d4a3772a632b14e7d313fa231233a3f1dbb25
Jul 23 15:48:35.815015 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.814921    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sq4qw" event={"ID":"51b14893-b696-45a2-8d6e-78cf129dd263","Type":"ContainerStarted","Data":"d3a64e3906ead25e157ba7af3084f9342aa790f6dea7965471a8254ef5420043"}
Jul 23 15:48:35.817975 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.817939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t82bm" event={"ID":"1bddfe96-b526-4de7-981b-de1f84b81b3c","Type":"ContainerStarted","Data":"d46321ef5dfb00a9102e10e88b8d4a3772a632b14e7d313fa231233a3f1dbb25"}
Jul 23 15:48:35.820311 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.820273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz59x" event={"ID":"2dae43a1-ca00-4cf2-b8c0-ba84573a868a","Type":"ContainerStarted","Data":"f96654d6167783d5f6f4e973640d4db8de9037eeee68ae55678fbc0ba8120797"}
Jul 23 15:48:35.821574 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.821552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gdwzk" event={"ID":"8444dfcc-a56d-4220-86eb-2f7b56291263","Type":"ContainerStarted","Data":"398cf6ce56380684bbf2ff11b5437ab509370145e60a2cc558b111f6ac9e2eb6"}
Jul 23 15:48:35.822927 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.822898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vvcgh" event={"ID":"f3caef45-ca17-4dd6-b51a-18b82c00b863","Type":"ContainerStarted","Data":"8fd74568038a0e5a8672aa67f5b7388da35a6b44b234da95d0e973cd9b1c3153"}
Jul 23 15:48:35.824252 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.824226    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pnnqq" event={"ID":"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7","Type":"ContainerStarted","Data":"10ba7015a0bb19db55b592e095fe778e560af044443ad16eb5c8da06ff9d438a"}
Jul 23 15:48:35.825617 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.825583    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-skg42" event={"ID":"fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9","Type":"ContainerStarted","Data":"f35b34b0ba87270d8cded1b192f3415e4d9db9dae0835f8934408cc20a4aeddc"}
Jul 23 15:48:35.826682 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:35.826659    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf" event={"ID":"f05ff6b4-6863-4fdb-974e-9be9807c132a","Type":"ContainerStarted","Data":"8955a4d19f47811770e269b7c7920c9774be5a2f43b577910c03a2cea8f36247"}
Jul 23 15:48:35.832627 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:35.832602    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01e18b9_f07e_494f_9a2d_50f2f70471de.slice/crio-e4f6133686cacc82f8b34df9aa31221e456e8453db8cb93c0611bd559ca0d01f WatchSource:0}: Error finding container e4f6133686cacc82f8b34df9aa31221e456e8453db8cb93c0611bd559ca0d01f: Status 404 returned error can't find the container with id e4f6133686cacc82f8b34df9aa31221e456e8453db8cb93c0611bd559ca0d01f
Jul 23 15:48:35.859881 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:48:35.859856    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c061e960ca96a1ef3107eae942918a9.slice/crio-e7dd06a123a0cddfcff469bf73055b6583a18e1124e7b1c2c6ea60be40760eca WatchSource:0}: Error finding container e7dd06a123a0cddfcff469bf73055b6583a18e1124e7b1c2c6ea60be40760eca: Status 404 returned error can't find the container with id e7dd06a123a0cddfcff469bf73055b6583a18e1124e7b1c2c6ea60be40760eca
Jul 23 15:48:36.416459 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.416415    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:36.417053 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.416488    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:36.417053 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.416549    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:36.417053 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.416662    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:36.417053 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.416723    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs podName:46c729a0-b515-4581-9a67-1ee16d3649ab nodeName:}" failed. No retries permitted until 2026-07-23 15:48:38.416705285 +0000 UTC m=+6.042239760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs") pod "network-metrics-daemon-rghs4" (UID: "46c729a0-b515-4581-9a67-1ee16d3649ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:36.417268 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.417125    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:36.417268 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.417174    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret podName:dffd5fde-6708-475a-ae48-e3f9b983009a nodeName:}" failed. No retries permitted until 2026-07-23 15:48:38.417159023 +0000 UTC m=+6.042693499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret") pod "global-pull-secret-syncer-glxtl" (UID: "dffd5fde-6708-475a-ae48-e3f9b983009a") : object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:36.417268 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.417249    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 15:48:36.417268 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.417262    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 15:48:36.417458 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.417274    2555 projected.go:196] Error preparing data for projected volume kube-api-access-q7mx8 for pod openshift-network-diagnostics/network-check-target-zr2d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:36.417458 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.417304    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8 podName:d443f735-cecc-4b70-95e8-3bce5625a11c nodeName:}" failed. No retries permitted until 2026-07-23 15:48:38.417294518 +0000 UTC m=+6.042828985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7mx8" (UniqueName: "kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8") pod "network-check-target-zr2d7" (UID: "d443f735-cecc-4b70-95e8-3bce5625a11c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:36.734410 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.734272    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 15:43:34 +0000 UTC" deadline="2028-01-09 07:35:16.180316103 +0000 UTC"
Jul 23 15:48:36.734410 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.734331    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12831h46m39.445988455s"
Jul 23 15:48:36.806051 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.805956    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:36.806182 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.806071    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:36.806578 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.806446    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:36.806578 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:36.806536    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:36.836257 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.836224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"e4f6133686cacc82f8b34df9aa31221e456e8453db8cb93c0611bd559ca0d01f"}
Jul 23 15:48:36.848464 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.847906    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" event={"ID":"d0b8168cb74fd246a62dfc8f2491ac00","Type":"ContainerStarted","Data":"1ac1cac6f8e4d4c9251eb4806d94a184d88a8b31b45a367b91768c70168e74ae"}
Jul 23 15:48:36.857682 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.857584    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" event={"ID":"6c061e960ca96a1ef3107eae942918a9","Type":"ContainerStarted","Data":"e7dd06a123a0cddfcff469bf73055b6583a18e1124e7b1c2c6ea60be40760eca"}
Jul 23 15:48:36.864228 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:36.864179    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-121.ec2.internal" podStartSLOduration=2.864162378 podStartE2EDuration="2.864162378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:48:36.863203927 +0000 UTC m=+4.488738425" watchObservedRunningTime="2026-07-23 15:48:36.864162378 +0000 UTC m=+4.489696853"
Jul 23 15:48:37.804274 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:37.803695    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:37.804274 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:37.803841    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:38.433198    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:38.433245    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:38.433285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.433477    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.433499    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.433512    2555 projected.go:196] Error preparing data for projected volume kube-api-access-q7mx8 for pod openshift-network-diagnostics/network-check-target-zr2d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.433567    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8 podName:d443f735-cecc-4b70-95e8-3bce5625a11c nodeName:}" failed. No retries permitted until 2026-07-23 15:48:42.433549207 +0000 UTC m=+10.059083664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7mx8" (UniqueName: "kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8") pod "network-check-target-zr2d7" (UID: "d443f735-cecc-4b70-95e8-3bce5625a11c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.433590    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.433627    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs podName:46c729a0-b515-4581-9a67-1ee16d3649ab nodeName:}" failed. No retries permitted until 2026-07-23 15:48:42.433616689 +0000 UTC m=+10.059151160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs") pod "network-metrics-daemon-rghs4" (UID: "46c729a0-b515-4581-9a67-1ee16d3649ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.433659    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:38.433837 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.433721    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret podName:dffd5fde-6708-475a-ae48-e3f9b983009a nodeName:}" failed. No retries permitted until 2026-07-23 15:48:42.433705417 +0000 UTC m=+10.059239882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret") pod "global-pull-secret-syncer-glxtl" (UID: "dffd5fde-6708-475a-ae48-e3f9b983009a") : object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:38.803963 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:38.803844    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:38.803963 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:38.803853    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:38.804163 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.803991    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:38.804163 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:38.804052    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:39.804357 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:39.804323    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:39.804796 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:39.804477    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:40.804087 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:40.804023    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:40.804273 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:40.804196    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:40.804589 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:40.804422    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:40.804589 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:40.804541    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:41.804593 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:41.804560    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:41.805068 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:41.804705    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:42.463135    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:42.463186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:42.463236    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.463260    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.463325    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs podName:46c729a0-b515-4581-9a67-1ee16d3649ab nodeName:}" failed. No retries permitted until 2026-07-23 15:48:50.463305599 +0000 UTC m=+18.088840068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs") pod "network-metrics-daemon-rghs4" (UID: "46c729a0-b515-4581-9a67-1ee16d3649ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.463349    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.463365    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.463376    2555 projected.go:196] Error preparing data for projected volume kube-api-access-q7mx8 for pod openshift-network-diagnostics/network-check-target-zr2d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.463416    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8 podName:d443f735-cecc-4b70-95e8-3bce5625a11c nodeName:}" failed. No retries permitted until 2026-07-23 15:48:50.463404272 +0000 UTC m=+18.088938744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7mx8" (UniqueName: "kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8") pod "network-check-target-zr2d7" (UID: "d443f735-cecc-4b70-95e8-3bce5625a11c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.463469    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:42.463525 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.463498    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret podName:dffd5fde-6708-475a-ae48-e3f9b983009a nodeName:}" failed. No retries permitted until 2026-07-23 15:48:50.463489397 +0000 UTC m=+18.089023868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret") pod "global-pull-secret-syncer-glxtl" (UID: "dffd5fde-6708-475a-ae48-e3f9b983009a") : object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:42.804945 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:42.804870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:42.805346 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.804970    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:42.805346 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:42.805329    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:42.805458 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:42.805411    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:43.803712 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:43.803676    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:43.803889 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:43.803815    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:44.804427 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:44.804394    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:44.804881 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:44.804440    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:44.804881 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:44.804530    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:44.804881 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:44.804658    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:45.803948 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:45.803903    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:45.804118 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:45.804051    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:46.806586 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:46.806557    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:46.807107 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:46.806671    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:46.807107 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:46.806765    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:46.807107 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:46.806865    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:47.804228 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:47.804189    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:47.804382 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:47.804308    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:48.806850 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:48.806820    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:48.807248 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:48.806830    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:48.807248 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:48.806929    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:48.807248 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:48.807008    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:49.804113 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:49.804036    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:49.804338 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:49.804161    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:50.529437 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:50.529402    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:50.529453    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:50.529492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.529576    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.529615    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.529624    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.529640    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs podName:46c729a0-b515-4581-9a67-1ee16d3649ab nodeName:}" failed. No retries permitted until 2026-07-23 15:49:06.529623492 +0000 UTC m=+34.155157958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs") pod "network-metrics-daemon-rghs4" (UID: "46c729a0-b515-4581-9a67-1ee16d3649ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.529680    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret podName:dffd5fde-6708-475a-ae48-e3f9b983009a nodeName:}" failed. No retries permitted until 2026-07-23 15:49:06.529662793 +0000 UTC m=+34.155197246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret") pod "global-pull-secret-syncer-glxtl" (UID: "dffd5fde-6708-475a-ae48-e3f9b983009a") : object "kube-system"/"original-pull-secret" not registered
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.529632    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.529706    2555 projected.go:196] Error preparing data for projected volume kube-api-access-q7mx8 for pod openshift-network-diagnostics/network-check-target-zr2d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:50.529893 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.529767    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8 podName:d443f735-cecc-4b70-95e8-3bce5625a11c nodeName:}" failed. No retries permitted until 2026-07-23 15:49:06.529742842 +0000 UTC m=+34.155277315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7mx8" (UniqueName: "kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8") pod "network-check-target-zr2d7" (UID: "d443f735-cecc-4b70-95e8-3bce5625a11c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:48:50.803653 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:50.803581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:50.803827 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:50.803581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:50.803827 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.803700    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:50.803827 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:50.803769    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:51.803819 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:51.803777    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:51.804222 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:51.803909    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:52.804738 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:52.804705    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:52.805178 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:52.804794    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:52.805178 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:52.804814    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:52.805178 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:52.804896    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:53.804166 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:53.804142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:53.804278 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:53.804260    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:54.807027 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.806975    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:54.807560 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:54.807071    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:54.807560 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.807096    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:54.807560 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:54.807145    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:54.901218 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.901121    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:48:54.901590 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.901572    2555 generic.go:356] "Generic (PLEG): container finished" podID="e01e18b9-f07e-494f-9a2d-50f2f70471de" containerID="b1f54520810e9b95f59c7307f30594f3d2ee68f87abfdcd54388b8bd65e07eb7" exitCode=1
Jul 23 15:48:54.901668 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.901636    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerDied","Data":"b1f54520810e9b95f59c7307f30594f3d2ee68f87abfdcd54388b8bd65e07eb7"}
Jul 23 15:48:54.901725 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.901709    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"c85a2000272c8d8d956e91872e0a1f2c83ec9437d485db4473384ea8a31488de"}
Jul 23 15:48:54.903245 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.903220    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sq4qw" event={"ID":"51b14893-b696-45a2-8d6e-78cf129dd263","Type":"ContainerStarted","Data":"dfada9362c1674201660c631c0f6e46b28d6e1a173d8d9fa53280ac87f31c7df"}
Jul 23 15:48:54.904640 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.904607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t82bm" event={"ID":"1bddfe96-b526-4de7-981b-de1f84b81b3c","Type":"ContainerStarted","Data":"a8c079e41e8610808f5f485d75e715f943b1d203fcacebc9605cf909f8d92041"}
Jul 23 15:48:54.906098 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.906079    2555 generic.go:356] "Generic (PLEG): container finished" podID="2dae43a1-ca00-4cf2-b8c0-ba84573a868a" containerID="b1fb9a42980d7e48a67494a8d374f3ca90743f3f547cc6a41be4d917fdc55349" exitCode=0
Jul 23 15:48:54.906179 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.906144    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz59x" event={"ID":"2dae43a1-ca00-4cf2-b8c0-ba84573a868a","Type":"ContainerDied","Data":"b1fb9a42980d7e48a67494a8d374f3ca90743f3f547cc6a41be4d917fdc55349"}
Jul 23 15:48:54.907616 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.907592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gdwzk" event={"ID":"8444dfcc-a56d-4220-86eb-2f7b56291263","Type":"ContainerStarted","Data":"262514273e33e7aa1d72fe61912037e573697ba4e8abb5158245ae37a45b5f90"}
Jul 23 15:48:54.909153 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.909131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vvcgh" event={"ID":"f3caef45-ca17-4dd6-b51a-18b82c00b863","Type":"ContainerStarted","Data":"f399d5e6bef323ad62ec53642def6c9647f96af70f2e0824be155527695967b1"}
Jul 23 15:48:54.910596 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.910573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pnnqq" event={"ID":"d96f9a1a-32c4-4c5c-bd5c-0c039d192cb7","Type":"ContainerStarted","Data":"3dd2c5ce7781403ea449d265b9defe169b2d9fbac1cdfdcf2145d6b765c8be39"}
Jul 23 15:48:54.912042 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.912019    2555 generic.go:356] "Generic (PLEG): container finished" podID="6c061e960ca96a1ef3107eae942918a9" containerID="dd6a94e8a4ba0d21f39081084e53f73f29c86d3f8e21658baa7cffad2823f244" exitCode=0
Jul 23 15:48:54.912114 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.912094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" event={"ID":"6c061e960ca96a1ef3107eae942918a9","Type":"ContainerDied","Data":"dd6a94e8a4ba0d21f39081084e53f73f29c86d3f8e21658baa7cffad2823f244"}
Jul 23 15:48:54.913454 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.913429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-skg42" event={"ID":"fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9","Type":"ContainerStarted","Data":"74bca95e76bf7df2ab5a1e9c41f1efd8cd714d51684e046bcbf974e900f7ae05"}
Jul 23 15:48:54.914775 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.914743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf" event={"ID":"f05ff6b4-6863-4fdb-974e-9be9807c132a","Type":"ContainerStarted","Data":"130da06b9b3b1fa7ae8b47f9b283545eed91e5b503c2969d509ac39a4de3acf1"}
Jul 23 15:48:54.919285 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.919251    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sq4qw" podStartSLOduration=5.014178661 podStartE2EDuration="22.919239986s" totalImagesPullingTime="17.905061325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:48:54.919095168 +0000 UTC m=+22.544629644" watchObservedRunningTime="2026-07-23 15:48:54.919239986 +0000 UTC m=+22.544774460"
Jul 23 15:48:54.931614 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.931579    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t82bm" podStartSLOduration=5.061746583 podStartE2EDuration="22.931568039s" totalImagesPullingTime="17.869821456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:48:54.931560672 +0000 UTC m=+22.557095145" watchObservedRunningTime="2026-07-23 15:48:54.931568039 +0000 UTC m=+22.557102513"
Jul 23 15:48:54.945173 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:54.945140    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-skg42" podStartSLOduration=5.04024064 podStartE2EDuration="22.945129927s" totalImagesPullingTime="17.904889287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:48:54.944667274 +0000 UTC m=+22.570201749" watchObservedRunningTime="2026-07-23 15:48:54.945129927 +0000 UTC m=+22.570664401"
Jul 23 15:48:55.006268 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.006232    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gdwzk" podStartSLOduration=4.685937177 podStartE2EDuration="23.00621974s" totalImagesPullingTime="18.320282563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:48:55.005816453 +0000 UTC m=+22.631350928" watchObservedRunningTime="2026-07-23 15:48:55.00621974 +0000 UTC m=+22.631754217"
Jul 23 15:48:55.042567 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.042530    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pnnqq" podStartSLOduration=4.363923282 podStartE2EDuration="23.042517043s" totalImagesPullingTime="18.678593761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:48:55.042308255 +0000 UTC m=+22.667842730" watchObservedRunningTime="2026-07-23 15:48:55.042517043 +0000 UTC m=+22.668051517"
Jul 23 15:48:55.058175 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.057939    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vvcgh" podStartSLOduration=12.47540783 podStartE2EDuration="23.057925796s" totalImagesPullingTime="10.582517966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:48:55.057498822 +0000 UTC m=+22.683033298" watchObservedRunningTime="2026-07-23 15:48:55.057925796 +0000 UTC m=+22.683460271"
Jul 23 15:48:55.803990 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.803962    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:55.804202 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:55.804084    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:55.918622 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.918555    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" event={"ID":"6c061e960ca96a1ef3107eae942918a9","Type":"ContainerStarted","Data":"769561fe4bc190870cdfb6a141b5675a5018ca47a3d948207595cc7baa03b53e"}
Jul 23 15:48:55.921877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.921857    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:48:55.922372 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.922346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"e82f75192691fde6d8ded7c7d23aa40834495e574445d28c7ba86ad65c622a01"}
Jul 23 15:48:55.922474 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.922427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"e3a0165ed5d371096428153d67ec58da63fcb4729919a14b562bd03266f99d28"}
Jul 23 15:48:55.922539 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.922481    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"fff82c19dd91558e31c5e615d54474506fe24ede62b1f4c69288c6b32a5787fe"}
Jul 23 15:48:55.922539 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.922526    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"2b869ebe6ec583f9293a48772953222eec9c7d1eb5cac05bb9f2960b3ca561c8"}
Jul 23 15:48:55.935202 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:55.935152    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-121.ec2.internal" podStartSLOduration=21.935137332 podStartE2EDuration="21.935137332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:48:55.934725475 +0000 UTC m=+23.560259949" watchObservedRunningTime="2026-07-23 15:48:55.935137332 +0000 UTC m=+23.560671807"
Jul 23 15:48:56.071563 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:56.071541    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 15:48:56.756971 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:56.756867    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T15:48:56.071559377Z","UUID":"0074e7cd-3ac7-4d09-92b4-b3715d8496a9","Handler":null,"Name":"","Endpoint":""}
Jul 23 15:48:56.758333 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:56.758308    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 15:48:56.758462 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:56.758348    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 15:48:56.804331 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:56.804303    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:56.804497 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:56.804416    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:56.804563 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:56.804540    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:56.804663 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:56.804642    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:56.926402 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:56.926366    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf" event={"ID":"f05ff6b4-6863-4fdb-974e-9be9807c132a","Type":"ContainerStarted","Data":"325e1cb17a47f0d993fa420d935a35ca82a4eadc2465ff72170922daafc070c0"}
Jul 23 15:48:57.804121 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:57.803847    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:57.804267 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:57.804193    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:57.932004 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:57.931981    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:48:57.932453 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:57.932432    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"bbdbb978536718d1d67318ed06a266d372d1edd08bb80f93dc48e06b7d60e2f7"}
Jul 23 15:48:58.135289 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:58.135210    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:58.135836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:58.135815    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:58.804189 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:58.804159    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:48:58.804392 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:58.804285    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:48:58.804723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:58.804699    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:48:58.804862 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:58.804826    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:48:58.934418 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:58.934376    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:58.934889 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:58.934852    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-skg42"
Jul 23 15:48:59.804326 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:59.804112    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:48:59.804449 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:48:59.804348    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:48:59.937847 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:59.937815    2555 generic.go:356] "Generic (PLEG): container finished" podID="2dae43a1-ca00-4cf2-b8c0-ba84573a868a" containerID="e358243491523c7abc8fc64c7e094cfc433975a30a9c7bff049d87b02a778ce6" exitCode=0
Jul 23 15:48:59.938492 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:59.937897    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz59x" event={"ID":"2dae43a1-ca00-4cf2-b8c0-ba84573a868a","Type":"ContainerDied","Data":"e358243491523c7abc8fc64c7e094cfc433975a30a9c7bff049d87b02a778ce6"}
Jul 23 15:48:59.941547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:59.941532    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:48:59.941937 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:59.941906    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"b3b032f608fa85eb3250f8fd199ee44b3d4ff148c4eeeb0674b14be4ef326fd1"}
Jul 23 15:48:59.945296 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:48:59.943510    2555 scope.go:117] "RemoveContainer" containerID="b1f54520810e9b95f59c7307f30594f3d2ee68f87abfdcd54388b8bd65e07eb7"
Jul 23 15:49:00.807010 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.806510    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:00.807010 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:00.806636    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:49:00.807010 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.806697    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:00.807010 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:00.806814    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:49:00.946180 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.946146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf" event={"ID":"f05ff6b4-6863-4fdb-974e-9be9807c132a","Type":"ContainerStarted","Data":"7d2e099cde7670bab55dfd7a73c70bc39f641a5e4901d1e6413da39a4636b6a3"}
Jul 23 15:49:00.949847 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.949827    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:49:00.950175 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.950153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" event={"ID":"e01e18b9-f07e-494f-9a2d-50f2f70471de","Type":"ContainerStarted","Data":"46a902289ad7ea4e231781d6e5a89eea7121fdcb3f07cb4682946ccbd11952a4"}
Jul 23 15:49:00.950444 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.950420    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:49:00.950549 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.950493    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:49:00.950549 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.950543    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:49:00.964272 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.964234    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cr2kf" podStartSLOduration=5.000904415 podStartE2EDuration="28.964217385s" totalImagesPullingTime="23.96331297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:00.964171562 +0000 UTC m=+28.589706037" watchObservedRunningTime="2026-07-23 15:49:00.964217385 +0000 UTC m=+28.589751862"
Jul 23 15:49:00.965314 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.965294    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:49:00.965603 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.965590    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:49:00.991030 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:00.990996    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g" podStartSLOduration=10.353390725 podStartE2EDuration="28.990984384s" totalImagesPullingTime="18.637593659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:00.990824963 +0000 UTC m=+28.616359438" watchObservedRunningTime="2026-07-23 15:49:00.990984384 +0000 UTC m=+28.616518858"
Jul 23 15:49:01.468742 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:01.468553    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-glxtl"]
Jul 23 15:49:01.468876 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:01.468830    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:01.468933 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:01.468911    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:49:01.472575 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:01.472545    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr2d7"]
Jul 23 15:49:01.472699 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:01.472655    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:01.472790 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:01.472767    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:49:01.473170 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:01.473149    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rghs4"]
Jul 23 15:49:01.473272 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:01.473259    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:01.473397 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:01.473377    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:49:01.954576 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:01.954488    2555 generic.go:356] "Generic (PLEG): container finished" podID="2dae43a1-ca00-4cf2-b8c0-ba84573a868a" containerID="b31788708cf13a698f92860e3392227e1cbdad9788b9cae7a02766b6bd4f7b32" exitCode=0
Jul 23 15:49:01.954975 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:01.954576    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz59x" event={"ID":"2dae43a1-ca00-4cf2-b8c0-ba84573a868a","Type":"ContainerDied","Data":"b31788708cf13a698f92860e3392227e1cbdad9788b9cae7a02766b6bd4f7b32"}
Jul 23 15:49:02.804731 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:02.804709    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:02.804834 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:02.804797    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:02.804834 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:02.804814    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:49:02.804925 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:02.804903    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:49:02.959909 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:02.959875    2555 generic.go:356] "Generic (PLEG): container finished" podID="2dae43a1-ca00-4cf2-b8c0-ba84573a868a" containerID="a09a41454f454964f2b78a4d6ac1b926398c98163222f3069a0c56a2d34f93d4" exitCode=0
Jul 23 15:49:02.960708 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:02.960679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz59x" event={"ID":"2dae43a1-ca00-4cf2-b8c0-ba84573a868a","Type":"ContainerDied","Data":"a09a41454f454964f2b78a4d6ac1b926398c98163222f3069a0c56a2d34f93d4"}
Jul 23 15:49:03.803916 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:03.803847    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:03.804065 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:03.803979    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:49:04.804476 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:04.804441    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:04.805078 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:04.804571    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:49:04.805078 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:04.804631    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:04.805078 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:04.804769    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:49:05.803823 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:05.803788    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:05.804003 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:05.803903    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:49:06.560983 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:06.560908    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:06.560983 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:06.560946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:06.560987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.561060    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.561077    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.561091    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.561103    2555 projected.go:196] Error preparing data for projected volume kube-api-access-q7mx8 for pod openshift-network-diagnostics/network-check-target-zr2d7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.561115    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.561128    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs podName:46c729a0-b515-4581-9a67-1ee16d3649ab nodeName:}" failed. No retries permitted until 2026-07-23 15:49:38.56110873 +0000 UTC m=+66.186643194 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs") pod "network-metrics-daemon-rghs4" (UID: "46c729a0-b515-4581-9a67-1ee16d3649ab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.561146    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8 podName:d443f735-cecc-4b70-95e8-3bce5625a11c nodeName:}" failed. No retries permitted until 2026-07-23 15:49:38.561137678 +0000 UTC m=+66.186672133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7mx8" (UniqueName: "kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8") pod "network-check-target-zr2d7" (UID: "d443f735-cecc-4b70-95e8-3bce5625a11c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 15:49:06.561532 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.561172    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret podName:dffd5fde-6708-475a-ae48-e3f9b983009a nodeName:}" failed. No retries permitted until 2026-07-23 15:49:38.561155842 +0000 UTC m=+66.186690315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret") pod "global-pull-secret-syncer-glxtl" (UID: "dffd5fde-6708-475a-ae48-e3f9b983009a") : object "kube-system"/"original-pull-secret" not registered
Jul 23 15:49:06.804005 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:06.803977    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:06.804167 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.804108    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zr2d7" podUID="d443f735-cecc-4b70-95e8-3bce5625a11c"
Jul 23 15:49:06.804232 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:06.804165    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:06.804299 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:06.804277    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glxtl" podUID="dffd5fde-6708-475a-ae48-e3f9b983009a"
Jul 23 15:49:07.804315 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:07.804286    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:07.804612 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:07.804395    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rghs4" podUID="46c729a0-b515-4581-9a67-1ee16d3649ab"
Jul 23 15:49:07.927450 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:07.927394    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gdwzk_8444dfcc-a56d-4220-86eb-2f7b56291263/dns-node-resolver/0.log"
Jul 23 15:49:08.169837 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.169811    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-121.ec2.internal" event="NodeReady"
Jul 23 15:49:08.169946 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.169936    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 15:49:08.207460 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.207364    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"]
Jul 23 15:49:08.209247 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.209228    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.211943 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.211927    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 23 15:49:08.212270 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.212249    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j9w79\""
Jul 23 15:49:08.212270 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.212267    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 23 15:49:08.212424 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.212407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 23 15:49:08.217974 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.217955    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 23 15:49:08.221968 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.221949    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lm22h"]
Jul 23 15:49:08.223877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.223863    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.226233 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.226181    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-94s22\""
Jul 23 15:49:08.226233 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.226230    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 15:49:08.226377 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.226258    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 15:49:08.227069 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.226935    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9dh8x"]
Jul 23 15:49:08.229026 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.229010    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"]
Jul 23 15:49:08.229111 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.229100    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.231326 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.231307    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmhs6\""
Jul 23 15:49:08.231395 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.231333    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 15:49:08.231534 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.231522    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 15:49:08.231664 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.231644    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 15:49:08.231862 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.231843    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 15:49:08.238154 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.238131    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lm22h"]
Jul 23 15:49:08.241248 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.241230    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9dh8x"]
Jul 23 15:49:08.243906 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.243888    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g4zf2"]
Jul 23 15:49:08.245883 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.245867    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g4zf2"
Jul 23 15:49:08.248218 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.248200    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 15:49:08.248649 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.248631    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jv7t4\""
Jul 23 15:49:08.248736 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.248663    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 15:49:08.248736 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.248677    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 15:49:08.258562 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.258543    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g4zf2"]
Jul 23 15:49:08.372351 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372315    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-config-volume\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.372351 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372354    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb77c59d-a9b7-40d0-85f6-c0005b97d735-crio-socket\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.372535 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372396    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzbn\" (UniqueName: \"kubernetes.io/projected/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-kube-api-access-mzzbn\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.372535 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372412    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.372535 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372426    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e4692c-174e-4e57-a7b0-7ec577cea880-cert\") pod \"ingress-canary-g4zf2\" (UID: \"10e4692c-174e-4e57-a7b0-7ec577cea880\") " pod="openshift-ingress-canary/ingress-canary-g4zf2"
Jul 23 15:49:08.372535 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372463    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-image-registry-private-configuration\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.372535 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372484    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-certificates\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.372701 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372530    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-tmp-dir\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.372701 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372572    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bngfp\" (UniqueName: \"kubernetes.io/projected/10e4692c-174e-4e57-a7b0-7ec577cea880-kube-api-access-bngfp\") pod \"ingress-canary-g4zf2\" (UID: \"10e4692c-174e-4e57-a7b0-7ec577cea880\") " pod="openshift-ingress-canary/ingress-canary-g4zf2"
Jul 23 15:49:08.372701 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372598    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcr9b\" (UniqueName: \"kubernetes.io/projected/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-api-access-lcr9b\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.372701 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb77c59d-a9b7-40d0-85f6-c0005b97d735-data-volume\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.372701 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372648    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.372701 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.372900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372711    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-installation-pull-secrets\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.372900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.372900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32ce7a51-fb39-4685-baa2-3ab6176d92d9-ca-trust-extracted\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.372900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-trusted-ca\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.372900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-bound-sa-token\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.372900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.372828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25547\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-kube-api-access-25547\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474093 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474015    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.474093 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474057    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32ce7a51-fb39-4685-baa2-3ab6176d92d9-ca-trust-extracted\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474093 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474073    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-trusted-ca\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474275 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.474224    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 15:49:08.474275 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474253    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-bound-sa-token\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474361 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.474280    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls podName:0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:08.974264319 +0000 UTC m=+36.599798775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls") pod "dns-default-lm22h" (UID: "0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8") : secret "dns-default-metrics-tls" not found
Jul 23 15:49:08.474361 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474303    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25547\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-kube-api-access-25547\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474361 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474349    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-config-volume\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.474500 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb77c59d-a9b7-40d0-85f6-c0005b97d735-crio-socket\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.474500 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzbn\" (UniqueName: \"kubernetes.io/projected/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-kube-api-access-mzzbn\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.474500 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474436    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474500 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e4692c-174e-4e57-a7b0-7ec577cea880-cert\") pod \"ingress-canary-g4zf2\" (UID: \"10e4692c-174e-4e57-a7b0-7ec577cea880\") " pod="openshift-ingress-canary/ingress-canary-g4zf2"
Jul 23 15:49:08.474500 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32ce7a51-fb39-4685-baa2-3ab6176d92d9-ca-trust-extracted\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474705 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474538    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-image-registry-private-configuration\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474705 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474571    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-certificates\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.474705 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474609    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-tmp-dir\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.474705 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474625    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb77c59d-a9b7-40d0-85f6-c0005b97d735-crio-socket\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.474705 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bngfp\" (UniqueName: \"kubernetes.io/projected/10e4692c-174e-4e57-a7b0-7ec577cea880-kube-api-access-bngfp\") pod \"ingress-canary-g4zf2\" (UID: \"10e4692c-174e-4e57-a7b0-7ec577cea880\") " pod="openshift-ingress-canary/ingress-canary-g4zf2"
Jul 23 15:49:08.474705 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.474677    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 15:49:08.474705 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcr9b\" (UniqueName: \"kubernetes.io/projected/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-api-access-lcr9b\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.474705 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.474693    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8dd7b7c-dmdrf: secret "image-registry-tls" not found
Jul 23 15:49:08.475098 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.474734    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls podName:32ce7a51-fb39-4685-baa2-3ab6176d92d9 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:08.974720429 +0000 UTC m=+36.600254883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls") pod "image-registry-69c8dd7b7c-dmdrf" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9") : secret "image-registry-tls" not found
Jul 23 15:49:08.475098 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb77c59d-a9b7-40d0-85f6-c0005b97d735-data-volume\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.475098 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.475098 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.475098 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474882    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-installation-pull-secrets\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.475098 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.474977    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-tmp-dir\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.475371 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.475160    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-certificates\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.475371 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.475210    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-trusted-ca\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.475371 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.475221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-config-volume\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.475371 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.475280    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 15:49:08.475371 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.475291    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb77c59d-a9b7-40d0-85f6-c0005b97d735-data-volume\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.475371 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.475324    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm podName:eb77c59d-a9b7-40d0-85f6-c0005b97d735 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:08.97531253 +0000 UTC m=+36.600847009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9dh8x" (UID: "eb77c59d-a9b7-40d0-85f6-c0005b97d735") : configmap "kube-rbac-proxy" not found
Jul 23 15:49:08.475371 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.475341    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:08.475627 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.475396    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls podName:eb77c59d-a9b7-40d0-85f6-c0005b97d735 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:08.975370878 +0000 UTC m=+36.600905344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dh8x" (UID: "eb77c59d-a9b7-40d0-85f6-c0005b97d735") : secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:08.478706 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.478685    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-installation-pull-secrets\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.478706 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.478696    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-image-registry-private-configuration\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.478860 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.478712    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e4692c-174e-4e57-a7b0-7ec577cea880-cert\") pod \"ingress-canary-g4zf2\" (UID: \"10e4692c-174e-4e57-a7b0-7ec577cea880\") " pod="openshift-ingress-canary/ingress-canary-g4zf2"
Jul 23 15:49:08.483128 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.483054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzbn\" (UniqueName: \"kubernetes.io/projected/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-kube-api-access-mzzbn\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.483604 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.483579    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bngfp\" (UniqueName: \"kubernetes.io/projected/10e4692c-174e-4e57-a7b0-7ec577cea880-kube-api-access-bngfp\") pod \"ingress-canary-g4zf2\" (UID: \"10e4692c-174e-4e57-a7b0-7ec577cea880\") " pod="openshift-ingress-canary/ingress-canary-g4zf2"
Jul 23 15:49:08.483946 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.483927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-bound-sa-token\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.484032 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.484016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25547\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-kube-api-access-25547\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.484261 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.484248    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcr9b\" (UniqueName: \"kubernetes.io/projected/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-api-access-lcr9b\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.553378 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.553360    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g4zf2"
Jul 23 15:49:08.679340 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.679313    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g4zf2"]
Jul 23 15:49:08.682398 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:08.682373    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e4692c_174e_4e57_a7b0_7ec577cea880.slice/crio-bcb4b6179e17e1ddc711699626e064d953de766b4bd805bfefe09fe1cf5dbd66 WatchSource:0}: Error finding container bcb4b6179e17e1ddc711699626e064d953de766b4bd805bfefe09fe1cf5dbd66: Status 404 returned error can't find the container with id bcb4b6179e17e1ddc711699626e064d953de766b4bd805bfefe09fe1cf5dbd66
Jul 23 15:49:08.776605 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.776542    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tnwvf"]
Jul 23 15:49:08.780030 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.780011    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.782517 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.782498    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 23 15:49:08.783331 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.783316    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 23 15:49:08.783414 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.783334    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 23 15:49:08.783564 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.783549    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cxp99\""
Jul 23 15:49:08.783634 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.783609    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 23 15:49:08.792370 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.792353    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tnwvf"]
Jul 23 15:49:08.803600 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.803556    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:08.803787 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.803771    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:08.806284 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.806264    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 15:49:08.806589 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.806340    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 15:49:08.806589 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.806391    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bn5p9\""
Jul 23 15:49:08.806589 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.806393    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 15:49:08.878026 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.878002    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/31db2c13-6a90-486d-bec0-43bf6473d8ea-signing-key\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.878107 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.878027    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhkv\" (UniqueName: \"kubernetes.io/projected/31db2c13-6a90-486d-bec0-43bf6473d8ea-kube-api-access-xfhkv\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.878107 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.878071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/31db2c13-6a90-486d-bec0-43bf6473d8ea-signing-cabundle\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.901484 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.901469    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t82bm_1bddfe96-b526-4de7-981b-de1f84b81b3c/node-ca/0.log"
Jul 23 15:49:08.978401 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.978379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/31db2c13-6a90-486d-bec0-43bf6473d8ea-signing-key\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.978501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.978411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhkv\" (UniqueName: \"kubernetes.io/projected/31db2c13-6a90-486d-bec0-43bf6473d8ea-kube-api-access-xfhkv\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.978501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.978461    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.978501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.978486    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:08.978652 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.978515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/31db2c13-6a90-486d-bec0-43bf6473d8ea-signing-cabundle\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.978652 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.978543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:08.978652 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978581    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 15:49:08.978652 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.978614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:08.978652 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978639    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm podName:eb77c59d-a9b7-40d0-85f6-c0005b97d735 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:09.978621949 +0000 UTC m=+37.604156404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9dh8x" (UID: "eb77c59d-a9b7-40d0-85f6-c0005b97d735") : configmap "kube-rbac-proxy" not found
Jul 23 15:49:08.978652 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978649    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:08.979037 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978693    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 15:49:08.979037 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978705    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8dd7b7c-dmdrf: secret "image-registry-tls" not found
Jul 23 15:49:08.979037 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978708    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls podName:eb77c59d-a9b7-40d0-85f6-c0005b97d735 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:09.978692924 +0000 UTC m=+37.604227391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dh8x" (UID: "eb77c59d-a9b7-40d0-85f6-c0005b97d735") : secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:08.979037 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978739    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls podName:32ce7a51-fb39-4685-baa2-3ab6176d92d9 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:09.978727841 +0000 UTC m=+37.604262308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls") pod "image-registry-69c8dd7b7c-dmdrf" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9") : secret "image-registry-tls" not found
Jul 23 15:49:08.979037 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978737    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 15:49:08.979037 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:08.978794    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls podName:0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:09.978785017 +0000 UTC m=+37.604319470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls") pod "dns-default-lm22h" (UID: "0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8") : secret "dns-default-metrics-tls" not found
Jul 23 15:49:08.979254 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.979236    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/31db2c13-6a90-486d-bec0-43bf6473d8ea-signing-cabundle\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.980215 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.980195    2555 generic.go:356] "Generic (PLEG): container finished" podID="2dae43a1-ca00-4cf2-b8c0-ba84573a868a" containerID="8d4fb60ddd0e8f4b26a49ecd2c71faac22b81b43a7bd2c11857984d4c055d379" exitCode=0
Jul 23 15:49:08.980300 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.980273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz59x" event={"ID":"2dae43a1-ca00-4cf2-b8c0-ba84573a868a","Type":"ContainerDied","Data":"8d4fb60ddd0e8f4b26a49ecd2c71faac22b81b43a7bd2c11857984d4c055d379"}
Jul 23 15:49:08.981495 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.981471    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g4zf2" event={"ID":"10e4692c-174e-4e57-a7b0-7ec577cea880","Type":"ContainerStarted","Data":"bcb4b6179e17e1ddc711699626e064d953de766b4bd805bfefe09fe1cf5dbd66"}
Jul 23 15:49:08.981758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.981734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/31db2c13-6a90-486d-bec0-43bf6473d8ea-signing-key\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:08.986610 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:08.986593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhkv\" (UniqueName: \"kubernetes.io/projected/31db2c13-6a90-486d-bec0-43bf6473d8ea-kube-api-access-xfhkv\") pod \"service-ca-58598695f7-tnwvf\" (UID: \"31db2c13-6a90-486d-bec0-43bf6473d8ea\") " pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:09.088827 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.088630    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-tnwvf"
Jul 23 15:49:09.200546 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.200522    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tnwvf"]
Jul 23 15:49:09.202961 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:09.202935    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31db2c13_6a90_486d_bec0_43bf6473d8ea.slice/crio-11f57f333845458450baf061b37b98d2306bbb8ecd936247ef4da083b4bbc9ef WatchSource:0}: Error finding container 11f57f333845458450baf061b37b98d2306bbb8ecd936247ef4da083b4bbc9ef: Status 404 returned error can't find the container with id 11f57f333845458450baf061b37b98d2306bbb8ecd936247ef4da083b4bbc9ef
Jul 23 15:49:09.804482 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.804446    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:09.810793 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.810767    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pzsj9\""
Jul 23 15:49:09.811286 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.811054    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 15:49:09.989112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.988678    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:09.989112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.988732    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:09.989112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.988800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:09.989112 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.988888    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 15:49:09.989112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.988903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:09.989112 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.988980    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm podName:eb77c59d-a9b7-40d0-85f6-c0005b97d735 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:11.988957989 +0000 UTC m=+39.614492464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9dh8x" (UID: "eb77c59d-a9b7-40d0-85f6-c0005b97d735") : configmap "kube-rbac-proxy" not found
Jul 23 15:49:09.989552 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.989200    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 15:49:09.989552 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.989217    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8dd7b7c-dmdrf: secret "image-registry-tls" not found
Jul 23 15:49:09.989552 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.989274    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls podName:32ce7a51-fb39-4685-baa2-3ab6176d92d9 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:11.9892588 +0000 UTC m=+39.614793255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls") pod "image-registry-69c8dd7b7c-dmdrf" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9") : secret "image-registry-tls" not found
Jul 23 15:49:09.989552 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.989364    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:09.989552 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.989539    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 15:49:09.989791 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.989615    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls podName:0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:11.989595267 +0000 UTC m=+39.615129740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls") pod "dns-default-lm22h" (UID: "0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8") : secret "dns-default-metrics-tls" not found
Jul 23 15:49:09.992017 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:09.989931    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls podName:eb77c59d-a9b7-40d0-85f6-c0005b97d735 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:11.989911287 +0000 UTC m=+39.615445752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dh8x" (UID: "eb77c59d-a9b7-40d0-85f6-c0005b97d735") : secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:09.994172 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.994140    2555 generic.go:356] "Generic (PLEG): container finished" podID="2dae43a1-ca00-4cf2-b8c0-ba84573a868a" containerID="a7d84d6469347a6cb2c2dc802a4623ec959867ec79c5df19eff7bd781915788c" exitCode=0
Jul 23 15:49:09.994803 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.994774    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz59x" event={"ID":"2dae43a1-ca00-4cf2-b8c0-ba84573a868a","Type":"ContainerDied","Data":"a7d84d6469347a6cb2c2dc802a4623ec959867ec79c5df19eff7bd781915788c"}
Jul 23 15:49:09.996886 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:09.996865    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-tnwvf" event={"ID":"31db2c13-6a90-486d-bec0-43bf6473d8ea","Type":"ContainerStarted","Data":"11f57f333845458450baf061b37b98d2306bbb8ecd936247ef4da083b4bbc9ef"}
Jul 23 15:49:11.002385 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:11.002324    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz59x" event={"ID":"2dae43a1-ca00-4cf2-b8c0-ba84573a868a","Type":"ContainerStarted","Data":"cd41ae13923c17f33d146559140dfae9eba352159c8673baa9238b68579261d2"}
Jul 23 15:49:11.031584 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:11.031386    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fz59x" podStartSLOduration=9.781779926 podStartE2EDuration="39.031364866s" totalImagesPullingTime="29.24958494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:11.030992453 +0000 UTC m=+38.656526941" watchObservedRunningTime="2026-07-23 15:49:11.031364866 +0000 UTC m=+38.656899338"
Jul 23 15:49:12.006239 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.006211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.006293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:12.006366    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:12.006385    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8dd7b7c-dmdrf: secret "image-registry-tls" not found
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.006397    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.006423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:12.006454    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls podName:32ce7a51-fb39-4685-baa2-3ab6176d92d9 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:16.006435863 +0000 UTC m=+43.631970315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls") pod "image-registry-69c8dd7b7c-dmdrf" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9") : secret "image-registry-tls" not found
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:12.006509    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:12.006537    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:12.006553    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls podName:0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:16.00653877 +0000 UTC m=+43.632073223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls") pod "dns-default-lm22h" (UID: "0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8") : secret "dns-default-metrics-tls" not found
Jul 23 15:49:12.006677 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:12.006590    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls podName:eb77c59d-a9b7-40d0-85f6-c0005b97d735 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:16.006573928 +0000 UTC m=+43.632108382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dh8x" (UID: "eb77c59d-a9b7-40d0-85f6-c0005b97d735") : secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:12.007168 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.006680    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g4zf2" event={"ID":"10e4692c-174e-4e57-a7b0-7ec577cea880","Type":"ContainerStarted","Data":"000e9baa2edcbfd3a4c71e71776bf08eb7dd401235b21f201b531bc83a733423"}
Jul 23 15:49:12.007168 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.006918    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb77c59d-a9b7-40d0-85f6-c0005b97d735-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:12.008070 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.008050    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-tnwvf" event={"ID":"31db2c13-6a90-486d-bec0-43bf6473d8ea","Type":"ContainerStarted","Data":"612c19a56ef3ffd1b1f94d8fc5ba4d1e0dc588bb21e16d4ea6650982955b41f0"}
Jul 23 15:49:12.022861 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.022820    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g4zf2" podStartSLOduration=1.179673853 podStartE2EDuration="4.022806656s" totalImagesPullingTime="2.843132803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:12.022080191 +0000 UTC m=+39.647614665" watchObservedRunningTime="2026-07-23 15:49:12.022806656 +0000 UTC m=+39.648341130"
Jul 23 15:49:12.036836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:12.036796    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-tnwvf" podStartSLOduration=1.713401745 podStartE2EDuration="4.036783111s" totalImagesPullingTime="2.323381366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:12.036453362 +0000 UTC m=+39.661987841" watchObservedRunningTime="2026-07-23 15:49:12.036783111 +0000 UTC m=+39.662317586"
Jul 23 15:49:16.033294 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:16.033262    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:16.033318    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:16.033399    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:16.033412    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8dd7b7c-dmdrf: secret "image-registry-tls" not found
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:16.033463    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:16.033408    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:16.033515    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls podName:32ce7a51-fb39-4685-baa2-3ab6176d92d9 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:24.033495127 +0000 UTC m=+51.659029585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls") pod "image-registry-69c8dd7b7c-dmdrf" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9") : secret "image-registry-tls" not found
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:16.033536    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls podName:0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:24.033527166 +0000 UTC m=+51.659061620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls") pod "dns-default-lm22h" (UID: "0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8") : secret "dns-default-metrics-tls" not found
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:16.033549    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:16.033646 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:16.033585    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls podName:eb77c59d-a9b7-40d0-85f6-c0005b97d735 nodeName:}" failed. No retries permitted until 2026-07-23 15:49:24.033572555 +0000 UTC m=+51.659107008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9dh8x" (UID: "eb77c59d-a9b7-40d0-85f6-c0005b97d735") : secret "insights-runtime-extractor-tls" not found
Jul 23 15:49:24.089025 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.088993    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:24.089025 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.089030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:24.089522 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.089169    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:24.092136 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.092108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb77c59d-a9b7-40d0-85f6-c0005b97d735-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9dh8x\" (UID: \"eb77c59d-a9b7-40d0-85f6-c0005b97d735\") " pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:24.092246 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.092193    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8-metrics-tls\") pod \"dns-default-lm22h\" (UID: \"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8\") " pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:24.092246 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.092200    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"image-registry-69c8dd7b7c-dmdrf\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") " pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:24.119304 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.119279    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:24.132556 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.132532    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:24.138686 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.138663    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9dh8x"
Jul 23 15:49:24.298594 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.298540    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"]
Jul 23 15:49:24.316836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.316779    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lm22h"]
Jul 23 15:49:24.319071 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:24.319047    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9096f8_4ae2_4bd9_9daa_9fe9499e87a8.slice/crio-2d5ea5e348f4485eea3ef535e6ffcaf1c6418e350f7376e9cc6304ac8d4ef6c5 WatchSource:0}: Error finding container 2d5ea5e348f4485eea3ef535e6ffcaf1c6418e350f7376e9cc6304ac8d4ef6c5: Status 404 returned error can't find the container with id 2d5ea5e348f4485eea3ef535e6ffcaf1c6418e350f7376e9cc6304ac8d4ef6c5
Jul 23 15:49:24.333324 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:24.333302    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9dh8x"]
Jul 23 15:49:24.336281 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:24.336254    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb77c59d_a9b7_40d0_85f6_c0005b97d735.slice/crio-cf7843afc2ff0143a3a75133440c83e2f202d48df204c165c107de1a11eac1ca WatchSource:0}: Error finding container cf7843afc2ff0143a3a75133440c83e2f202d48df204c165c107de1a11eac1ca: Status 404 returned error can't find the container with id cf7843afc2ff0143a3a75133440c83e2f202d48df204c165c107de1a11eac1ca
Jul 23 15:49:25.040940 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:25.040859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lm22h" event={"ID":"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8","Type":"ContainerStarted","Data":"2d5ea5e348f4485eea3ef535e6ffcaf1c6418e350f7376e9cc6304ac8d4ef6c5"}
Jul 23 15:49:25.042823 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:25.042720    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9dh8x" event={"ID":"eb77c59d-a9b7-40d0-85f6-c0005b97d735","Type":"ContainerStarted","Data":"cf7843afc2ff0143a3a75133440c83e2f202d48df204c165c107de1a11eac1ca"}
Jul 23 15:49:25.045776 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:25.044974    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf" event={"ID":"32ce7a51-fb39-4685-baa2-3ab6176d92d9","Type":"ContainerStarted","Data":"f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e"}
Jul 23 15:49:25.045776 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:25.045015    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf" event={"ID":"32ce7a51-fb39-4685-baa2-3ab6176d92d9","Type":"ContainerStarted","Data":"0a833edbd471b73355b4da59af8b27c27ef9316a599b018646baa70ee5cdc74e"}
Jul 23 15:49:25.045776 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:25.045714    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:25.068874 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:25.068620    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf" podStartSLOduration=59.068602363 podStartE2EDuration="59.068602363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:25.066333442 +0000 UTC m=+52.691867916" watchObservedRunningTime="2026-07-23 15:49:25.068602363 +0000 UTC m=+52.694136820"
Jul 23 15:49:27.053836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:27.053794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lm22h" event={"ID":"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8","Type":"ContainerStarted","Data":"dc5a589991daa6c6f2dda3203639274a6ac9008d34a82422436f39f1d5082324"}
Jul 23 15:49:30.065196 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:30.065158    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lm22h" event={"ID":"0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8","Type":"ContainerStarted","Data":"2cb1131a6b6bd7d754e662f467738cca2d441512cb95117cc8325aafe0e79fbe"}
Jul 23 15:49:30.065635 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:30.065300    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:30.066805 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:30.066774    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9dh8x" event={"ID":"eb77c59d-a9b7-40d0-85f6-c0005b97d735","Type":"ContainerStarted","Data":"ab3fa8aac1ff31e15372cdc3d12f439b4cf6029c0f177d27954b03dc22154d88"}
Jul 23 15:49:30.084773 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:30.084712    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lm22h" podStartSLOduration=19.903944844 podStartE2EDuration="22.084694564s" totalImagesPullingTime="2.18074972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:30.084046411 +0000 UTC m=+57.709580887" watchObservedRunningTime="2026-07-23 15:49:30.084694564 +0000 UTC m=+57.710229038"
Jul 23 15:49:31.742497 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.742469    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"]
Jul 23 15:49:31.847418 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.847392    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"]
Jul 23 15:49:31.847573 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.847512    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:31.851202 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.851180    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 23 15:49:31.851479 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.851465    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 15:49:31.851521 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.851482    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 15:49:31.852511 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.852491    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 15:49:31.861675 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.861654    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"]
Jul 23 15:49:31.861813 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.861798    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"
Jul 23 15:49:31.864315 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.864295    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 23 15:49:31.864407 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.864342    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tcwpr\""
Jul 23 15:49:31.882092 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.882063    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"]
Jul 23 15:49:31.882092 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.882094    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"]
Jul 23 15:49:31.882236 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.882105    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"]
Jul 23 15:49:31.882236 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.882119    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"]
Jul 23 15:49:31.882236 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.882142    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74d9fffb5-42whm"]
Jul 23 15:49:31.882236 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.882164    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:31.885024 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.885001    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 23 15:49:31.885120 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.885023    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 23 15:49:31.885185 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.885154    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 23 15:49:31.885185 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.885167    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 23 15:49:31.906358 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.906339    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74d9fffb5-42whm"]
Jul 23 15:49:31.906432 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.906406    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:31.955550 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955529    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-registry-tls\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:31.955658 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955559    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9b4b425-1de2-460a-a421-9550a9e3b83c-installation-pull-secrets\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:31.955658 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955584    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:31.955658 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955628    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf82995-1ae4-4021-8ee0-87d0c698ed96-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:31.955769 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955674    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhzh\" (UniqueName: \"kubernetes.io/projected/2702ff18-f825-4b9c-adc2-234b2410907b-kube-api-access-pjhzh\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:31.955769 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955694    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-hub\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:31.955769 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955715    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jng\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-kube-api-access-x7jng\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:31.955887 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955779    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ebde0313-d0c7-4c48-a513-ab46ecada7f2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75d7b7b548-2mxrx\" (UID: \"ebde0313-d0c7-4c48-a513-ab46ecada7f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"
Jul 23 15:49:31.955887 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-bound-sa-token\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:31.955887 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955824    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:31.955887 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955854    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2702ff18-f825-4b9c-adc2-234b2410907b-klusterlet-config\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:31.955887 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955870    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9b4b425-1de2-460a-a421-9550a9e3b83c-image-registry-private-configuration\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:31.956071 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955892    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-ca\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:31.956071 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2702ff18-f825-4b9c-adc2-234b2410907b-tmp\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:31.956071 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955932    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzq94\" (UniqueName: \"kubernetes.io/projected/0bf82995-1ae4-4021-8ee0-87d0c698ed96-kube-api-access-nzq94\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:31.956071 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955947    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9b4b425-1de2-460a-a421-9550a9e3b83c-registry-certificates\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:31.956071 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rrnd\" (UniqueName: \"kubernetes.io/projected/ebde0313-d0c7-4c48-a513-ab46ecada7f2-kube-api-access-2rrnd\") pod \"managed-serviceaccount-addon-agent-75d7b7b548-2mxrx\" (UID: \"ebde0313-d0c7-4c48-a513-ab46ecada7f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"
Jul 23 15:49:31.956071 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.955991    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9b4b425-1de2-460a-a421-9550a9e3b83c-ca-trust-extracted\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:31.956071 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:31.956023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9b4b425-1de2-460a-a421-9550a9e3b83c-trusted-ca\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.056643 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056564    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2702ff18-f825-4b9c-adc2-234b2410907b-tmp\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:32.056643 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056603    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzq94\" (UniqueName: \"kubernetes.io/projected/0bf82995-1ae4-4021-8ee0-87d0c698ed96-kube-api-access-nzq94\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.056643 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9b4b425-1de2-460a-a421-9550a9e3b83c-registry-certificates\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.056914 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056651    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rrnd\" (UniqueName: \"kubernetes.io/projected/ebde0313-d0c7-4c48-a513-ab46ecada7f2-kube-api-access-2rrnd\") pod \"managed-serviceaccount-addon-agent-75d7b7b548-2mxrx\" (UID: \"ebde0313-d0c7-4c48-a513-ab46ecada7f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"
Jul 23 15:49:32.056914 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056670    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9b4b425-1de2-460a-a421-9550a9e3b83c-ca-trust-extracted\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.056914 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056688    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9b4b425-1de2-460a-a421-9550a9e3b83c-trusted-ca\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.056914 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-registry-tls\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.056914 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056729    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9b4b425-1de2-460a-a421-9550a9e3b83c-installation-pull-secrets\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.056914 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056801    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.056914 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.056826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf82995-1ae4-4021-8ee0-87d0c698ed96-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.057242 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057012    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2702ff18-f825-4b9c-adc2-234b2410907b-tmp\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:32.057242 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhzh\" (UniqueName: \"kubernetes.io/projected/2702ff18-f825-4b9c-adc2-234b2410907b-kube-api-access-pjhzh\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:32.057242 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-hub\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.057242 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057139    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jng\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-kube-api-access-x7jng\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.057242 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9b4b425-1de2-460a-a421-9550a9e3b83c-ca-trust-extracted\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.057242 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ebde0313-d0c7-4c48-a513-ab46ecada7f2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75d7b7b548-2mxrx\" (UID: \"ebde0313-d0c7-4c48-a513-ab46ecada7f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"
Jul 23 15:49:32.057242 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-bound-sa-token\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.057569 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057376    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.057569 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057425    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2702ff18-f825-4b9c-adc2-234b2410907b-klusterlet-config\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:32.057569 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057453    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9b4b425-1de2-460a-a421-9550a9e3b83c-image-registry-private-configuration\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.057569 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf82995-1ae4-4021-8ee0-87d0c698ed96-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.057569 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.057492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-ca\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.058466 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.058436    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9b4b425-1de2-460a-a421-9550a9e3b83c-registry-certificates\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.058844 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.058803    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9b4b425-1de2-460a-a421-9550a9e3b83c-trusted-ca\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.060398 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.060374    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.060398 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.060394    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ebde0313-d0c7-4c48-a513-ab46ecada7f2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75d7b7b548-2mxrx\" (UID: \"ebde0313-d0c7-4c48-a513-ab46ecada7f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"
Jul 23 15:49:32.061148 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.061125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9b4b425-1de2-460a-a421-9550a9e3b83c-installation-pull-secrets\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.061281 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.061165    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.061281 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.061189    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-hub\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.061446 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.061430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2702ff18-f825-4b9c-adc2-234b2410907b-klusterlet-config\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:32.062096 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.062039    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-registry-tls\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.062318 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.062296    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0bf82995-1ae4-4021-8ee0-87d0c698ed96-ca\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.062918 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.062899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9b4b425-1de2-460a-a421-9550a9e3b83c-image-registry-private-configuration\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.065710 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.065685    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzq94\" (UniqueName: \"kubernetes.io/projected/0bf82995-1ae4-4021-8ee0-87d0c698ed96-kube-api-access-nzq94\") pod \"cluster-proxy-proxy-agent-6c6969d57-zzx5n\" (UID: \"0bf82995-1ae4-4021-8ee0-87d0c698ed96\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.066346 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.066324    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rrnd\" (UniqueName: \"kubernetes.io/projected/ebde0313-d0c7-4c48-a513-ab46ecada7f2-kube-api-access-2rrnd\") pod \"managed-serviceaccount-addon-agent-75d7b7b548-2mxrx\" (UID: \"ebde0313-d0c7-4c48-a513-ab46ecada7f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"
Jul 23 15:49:32.066650 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.066610    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhzh\" (UniqueName: \"kubernetes.io/projected/2702ff18-f825-4b9c-adc2-234b2410907b-kube-api-access-pjhzh\") pod \"klusterlet-addon-workmgr-759b68b45f-rbjqr\" (UID: \"2702ff18-f825-4b9c-adc2-234b2410907b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:32.067107 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.067075    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jng\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-kube-api-access-x7jng\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.067182 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.067124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9b4b425-1de2-460a-a421-9550a9e3b83c-bound-sa-token\") pod \"image-registry-74d9fffb5-42whm\" (UID: \"d9b4b425-1de2-460a-a421-9550a9e3b83c\") " pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.155868 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.155825    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:32.178115 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.178091    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"
Jul 23 15:49:32.190257 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.190231    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"
Jul 23 15:49:32.230856 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.230815    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:32.948355 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.948331    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74d9fffb5-42whm"]
Jul 23 15:49:32.951144 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:32.951112    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b4b425_1de2_460a_a421_9550a9e3b83c.slice/crio-2dad9b972076f0a9d88f349a477fc1a6bdd6cb21a2f789bb8ed59bebade56b61 WatchSource:0}: Error finding container 2dad9b972076f0a9d88f349a477fc1a6bdd6cb21a2f789bb8ed59bebade56b61: Status 404 returned error can't find the container with id 2dad9b972076f0a9d88f349a477fc1a6bdd6cb21a2f789bb8ed59bebade56b61
Jul 23 15:49:32.975701 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:32.975681    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-29d6g"
Jul 23 15:49:33.078557 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:33.078506    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74d9fffb5-42whm" event={"ID":"d9b4b425-1de2-460a-a421-9550a9e3b83c","Type":"ContainerStarted","Data":"2dad9b972076f0a9d88f349a477fc1a6bdd6cb21a2f789bb8ed59bebade56b61"}
Jul 23 15:49:33.208260 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:33.208233    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"]
Jul 23 15:49:33.209576 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:33.209558    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx"]
Jul 23 15:49:33.210809 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:33.210399    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebde0313_d0c7_4c48_a513_ab46ecada7f2.slice/crio-d0b461588111cd4bd17abdcc7daed41efb9cf5bd290cd51d86149399a7c15471 WatchSource:0}: Error finding container d0b461588111cd4bd17abdcc7daed41efb9cf5bd290cd51d86149399a7c15471: Status 404 returned error can't find the container with id d0b461588111cd4bd17abdcc7daed41efb9cf5bd290cd51d86149399a7c15471
Jul 23 15:49:33.211686 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:33.211536    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n"]
Jul 23 15:49:33.213361 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:33.213324    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bf82995_1ae4_4021_8ee0_87d0c698ed96.slice/crio-52b538d9243ff244699ed404e22bd173b966144cc75c483cdb472a21287f6c12 WatchSource:0}: Error finding container 52b538d9243ff244699ed404e22bd173b966144cc75c483cdb472a21287f6c12: Status 404 returned error can't find the container with id 52b538d9243ff244699ed404e22bd173b966144cc75c483cdb472a21287f6c12
Jul 23 15:49:34.088295 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:34.088257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9dh8x" event={"ID":"eb77c59d-a9b7-40d0-85f6-c0005b97d735","Type":"ContainerStarted","Data":"51a45033b6b87fa005cef824c10213130491ee2c7118bb7a2aeaebc2223b5b3c"}
Jul 23 15:49:34.091444 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:34.091137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n" event={"ID":"0bf82995-1ae4-4021-8ee0-87d0c698ed96","Type":"ContainerStarted","Data":"52b538d9243ff244699ed404e22bd173b966144cc75c483cdb472a21287f6c12"}
Jul 23 15:49:34.095878 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:34.095205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74d9fffb5-42whm" event={"ID":"d9b4b425-1de2-460a-a421-9550a9e3b83c","Type":"ContainerStarted","Data":"3fa39a82531e82c4aa97e93a4bf741e8f46858b41a54d4fef4cac7443231cdcb"}
Jul 23 15:49:34.095878 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:34.095869    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:34.096969 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:34.096913    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx" event={"ID":"ebde0313-d0c7-4c48-a513-ab46ecada7f2","Type":"ContainerStarted","Data":"d0b461588111cd4bd17abdcc7daed41efb9cf5bd290cd51d86149399a7c15471"}
Jul 23 15:49:34.098414 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:34.098361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr" event={"ID":"2702ff18-f825-4b9c-adc2-234b2410907b","Type":"ContainerStarted","Data":"1d999452906530e8c5e255cecd37933f97547cb3f3cac851fc08b629ffc9d95b"}
Jul 23 15:49:34.115553 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:34.115504    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74d9fffb5-42whm" podStartSLOduration=3.115485267 podStartE2EDuration="3.115485267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:34.114511803 +0000 UTC m=+61.740046280" watchObservedRunningTime="2026-07-23 15:49:34.115485267 +0000 UTC m=+61.741019740"
Jul 23 15:49:37.073544 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:37.073514    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lm22h"
Jul 23 15:49:38.620078 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.620038    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:38.620485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.620093    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:38.620485 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.620132    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:38.622945 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.622917    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 15:49:38.623730 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.623712    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 15:49:38.623820 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.623712    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 15:49:38.633145 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.633113    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c729a0-b515-4581-9a67-1ee16d3649ab-metrics-certs\") pod \"network-metrics-daemon-rghs4\" (UID: \"46c729a0-b515-4581-9a67-1ee16d3649ab\") " pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:38.633234 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.633217    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dffd5fde-6708-475a-ae48-e3f9b983009a-original-pull-secret\") pod \"global-pull-secret-syncer-glxtl\" (UID: \"dffd5fde-6708-475a-ae48-e3f9b983009a\") " pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:38.633234 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.633222    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 15:49:38.643377 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.643356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7mx8\" (UniqueName: \"kubernetes.io/projected/d443f735-cecc-4b70-95e8-3bce5625a11c-kube-api-access-q7mx8\") pod \"network-check-target-zr2d7\" (UID: \"d443f735-cecc-4b70-95e8-3bce5625a11c\") " pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:38.815258 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.815232    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bn5p9\""
Jul 23 15:49:38.817227 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.817206    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glxtl"
Jul 23 15:49:38.823418 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.823400    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:38.919528 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.919497    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pzsj9\""
Jul 23 15:49:38.927767 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:38.927732    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rghs4"
Jul 23 15:49:41.600197 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:41.600171    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rghs4"]
Jul 23 15:49:41.607059 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:41.607011    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c729a0_b515_4581_9a67_1ee16d3649ab.slice/crio-460caf37337a18901f9a26d8108ccc32d1e391886cadb428cd63ac9db76d5e2d WatchSource:0}: Error finding container 460caf37337a18901f9a26d8108ccc32d1e391886cadb428cd63ac9db76d5e2d: Status 404 returned error can't find the container with id 460caf37337a18901f9a26d8108ccc32d1e391886cadb428cd63ac9db76d5e2d
Jul 23 15:49:41.830515 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:41.830488    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-glxtl"]
Jul 23 15:49:41.831309 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:41.831288    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zr2d7"]
Jul 23 15:49:41.833164 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:41.833142    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffd5fde_6708_475a_ae48_e3f9b983009a.slice/crio-daeff995abf2e5b46b5c2bd380ad6feaef43a4ea0b34faaf9e1eb41e82b0376b WatchSource:0}: Error finding container daeff995abf2e5b46b5c2bd380ad6feaef43a4ea0b34faaf9e1eb41e82b0376b: Status 404 returned error can't find the container with id daeff995abf2e5b46b5c2bd380ad6feaef43a4ea0b34faaf9e1eb41e82b0376b
Jul 23 15:49:41.833630 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:49:41.833610    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd443f735_cecc_4b70_95e8_3bce5625a11c.slice/crio-32f4e26563b4a6ac0b973ef88b8439c195474b079ab49c214cc1d07803793324 WatchSource:0}: Error finding container 32f4e26563b4a6ac0b973ef88b8439c195474b079ab49c214cc1d07803793324: Status 404 returned error can't find the container with id 32f4e26563b4a6ac0b973ef88b8439c195474b079ab49c214cc1d07803793324
Jul 23 15:49:41.889443 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:41.889402    2555 patch_prober.go:28] interesting pod/image-registry-69c8dd7b7c-dmdrf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 23 15:49:41.889591 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:41.889497    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf" podUID="32ce7a51-fb39-4685-baa2-3ab6176d92d9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 15:49:42.131377 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.131301    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr" event={"ID":"2702ff18-f825-4b9c-adc2-234b2410907b","Type":"ContainerStarted","Data":"6c83b9f48c2a7318806f4468ebd127c856a80ec65d73ac49efd716d54a6278ca"}
Jul 23 15:49:42.131540 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.131483    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:42.133230 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.133208    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr"
Jul 23 15:49:42.133726 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.133698    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9dh8x" event={"ID":"eb77c59d-a9b7-40d0-85f6-c0005b97d735","Type":"ContainerStarted","Data":"eb6c8488ab24545b02b90d4a6f1df1c1ae2899cc15201d7590f235f2333fe969"}
Jul 23 15:49:42.135945 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.135924    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n" event={"ID":"0bf82995-1ae4-4021-8ee0-87d0c698ed96","Type":"ContainerStarted","Data":"93d42cd3aa810efb7c1304d51d914ef295806d992f7173ae204cb6168d7825d4"}
Jul 23 15:49:42.136030 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.135955    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n" event={"ID":"0bf82995-1ae4-4021-8ee0-87d0c698ed96","Type":"ContainerStarted","Data":"6ba0cc8d30a23a60f3ed8d263427d7653b0eca6b789c12468c7c5450ce9d055b"}
Jul 23 15:49:42.136030 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.135967    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n" event={"ID":"0bf82995-1ae4-4021-8ee0-87d0c698ed96","Type":"ContainerStarted","Data":"e6827673c4f76e5cf9270102f824e93881a19e98f333d4d307b5922033205a7e"}
Jul 23 15:49:42.137121 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.137097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rghs4" event={"ID":"46c729a0-b515-4581-9a67-1ee16d3649ab","Type":"ContainerStarted","Data":"460caf37337a18901f9a26d8108ccc32d1e391886cadb428cd63ac9db76d5e2d"}
Jul 23 15:49:42.138211 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.138183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr2d7" event={"ID":"d443f735-cecc-4b70-95e8-3bce5625a11c","Type":"ContainerStarted","Data":"32f4e26563b4a6ac0b973ef88b8439c195474b079ab49c214cc1d07803793324"}
Jul 23 15:49:42.139265 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.139218    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-glxtl" event={"ID":"dffd5fde-6708-475a-ae48-e3f9b983009a","Type":"ContainerStarted","Data":"daeff995abf2e5b46b5c2bd380ad6feaef43a4ea0b34faaf9e1eb41e82b0376b"}
Jul 23 15:49:42.140810 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.140787    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx" event={"ID":"ebde0313-d0c7-4c48-a513-ab46ecada7f2","Type":"ContainerStarted","Data":"93e1a66c89296ae83c67f74a98aca39df773101fa623cc1bc2847d5d8b022ed6"}
Jul 23 15:49:42.147096 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.147058    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-759b68b45f-rbjqr" podStartSLOduration=2.91496749 podStartE2EDuration="11.147046747s" totalImagesPullingTime="8.232079257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:42.14609223 +0000 UTC m=+69.771626706" watchObservedRunningTime="2026-07-23 15:49:42.147046747 +0000 UTC m=+69.772581222"
Jul 23 15:49:42.163388 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.163352    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9dh8x" podStartSLOduration=17.20583431 podStartE2EDuration="34.163339244s" totalImagesPullingTime="16.957504934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:49:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:42.162500873 +0000 UTC m=+69.788035348" watchObservedRunningTime="2026-07-23 15:49:42.163339244 +0000 UTC m=+69.788873718"
Jul 23 15:49:42.190179 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.190142    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75d7b7b548-2mxrx" podStartSLOduration=2.956176142 podStartE2EDuration="11.190130434s" totalImagesPullingTime="8.233954292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:42.189893769 +0000 UTC m=+69.815428244" watchObservedRunningTime="2026-07-23 15:49:42.190130434 +0000 UTC m=+69.815664909"
Jul 23 15:49:42.207298 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:42.207260    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c6969d57-zzx5n" podStartSLOduration=2.96019381 podStartE2EDuration="11.207247549s" totalImagesPullingTime="8.247053739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:42.206389588 +0000 UTC m=+69.831924078" watchObservedRunningTime="2026-07-23 15:49:42.207247549 +0000 UTC m=+69.832782023"
Jul 23 15:49:43.148544 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:43.147833    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rghs4" event={"ID":"46c729a0-b515-4581-9a67-1ee16d3649ab","Type":"ContainerStarted","Data":"2281a4ba2818359b408883dc94d3c65fd16b1d3b34d208e776b68f27158b533a"}
Jul 23 15:49:44.153699 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:44.153191    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rghs4" event={"ID":"46c729a0-b515-4581-9a67-1ee16d3649ab","Type":"ContainerStarted","Data":"675e258a979a978b60e54e46c7e242c2795968fcbf3ee32f5f64de1f1df30b12"}
Jul 23 15:49:44.169637 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:44.169593    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rghs4" podStartSLOduration=70.800430682 podStartE2EDuration="1m12.169576227s" totalImagesPullingTime="1.369145545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:44.168947689 +0000 UTC m=+71.794482166" watchObservedRunningTime="2026-07-23 15:49:44.169576227 +0000 UTC m=+71.795110699"
Jul 23 15:49:47.166438 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:47.166392    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zr2d7" event={"ID":"d443f735-cecc-4b70-95e8-3bce5625a11c","Type":"ContainerStarted","Data":"fea98509f458c91e1aa9953a97cef71e2a7838303a9dadf8091e6e2e9968dabc"}
Jul 23 15:49:47.166856 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:47.166515    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:49:47.167877 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:47.167853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-glxtl" event={"ID":"dffd5fde-6708-475a-ae48-e3f9b983009a","Type":"ContainerStarted","Data":"b18c97e1472a3a62fe07c1a034e11b95771f91aaa0c4178b230aeff9ebdc8ff1"}
Jul 23 15:49:47.181527 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:47.181481    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zr2d7" podStartSLOduration=70.222933535 podStartE2EDuration="1m15.181468919s" totalImagesPullingTime="4.958535384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:47.18092702 +0000 UTC m=+74.806461495" watchObservedRunningTime="2026-07-23 15:49:47.181468919 +0000 UTC m=+74.807003410"
Jul 23 15:49:47.195340 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:47.195303    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-glxtl" podStartSLOduration=70.236999399 podStartE2EDuration="1m15.195292443s" totalImagesPullingTime="4.958293044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:47.194673643 +0000 UTC m=+74.820208118" watchObservedRunningTime="2026-07-23 15:49:47.195292443 +0000 UTC m=+74.820826957"
Jul 23 15:49:51.887133 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:51.887102    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:56.114591 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:56.114463    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74d9fffb5-42whm"
Jul 23 15:49:56.901303 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:56.901256    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf" podUID="32ce7a51-fb39-4685-baa2-3ab6176d92d9" containerName="registry" containerID="cri-o://f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e" gracePeriod=30
Jul 23 15:49:57.132207 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.132187    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:57.200554 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.200496    2555 generic.go:356] "Generic (PLEG): container finished" podID="32ce7a51-fb39-4685-baa2-3ab6176d92d9" containerID="f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e" exitCode=0
Jul 23 15:49:57.200554 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.200535    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"
Jul 23 15:49:57.200554 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.200549    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf" event={"ID":"32ce7a51-fb39-4685-baa2-3ab6176d92d9","Type":"ContainerDied","Data":"f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e"}
Jul 23 15:49:57.200720 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.200592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8dd7b7c-dmdrf" event={"ID":"32ce7a51-fb39-4685-baa2-3ab6176d92d9","Type":"ContainerDied","Data":"0a833edbd471b73355b4da59af8b27c27ef9316a599b018646baa70ee5cdc74e"}
Jul 23 15:49:57.200720 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.200619    2555 scope.go:117] "RemoveContainer" containerID="f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e"
Jul 23 15:49:57.208032 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.208013    2555 scope.go:117] "RemoveContainer" containerID="f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e"
Jul 23 15:49:57.208294 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:49:57.208275    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e\": container with ID starting with f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e not found: ID does not exist" containerID="f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e"
Jul 23 15:49:57.208356 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.208302    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e"} err="failed to get container status \"f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e\": rpc error: code = NotFound desc = could not find container \"f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e\": container with ID starting with f7606f55a27b45dd04d9ffb47d43c43dd6a7d7e787461c93a6af51d01c0e032e not found: ID does not exist"
Jul 23 15:49:57.266481 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266461    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-certificates\") pod \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") "
Jul 23 15:49:57.266567 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266493    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-installation-pull-secrets\") pod \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") "
Jul 23 15:49:57.266567 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266511    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-trusted-ca\") pod \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") "
Jul 23 15:49:57.266567 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266540    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-bound-sa-token\") pod \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") "
Jul 23 15:49:57.266722 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266626    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") pod \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") "
Jul 23 15:49:57.266722 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266654    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25547\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-kube-api-access-25547\") pod \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") "
Jul 23 15:49:57.266722 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266704    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-image-registry-private-configuration\") pod \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") "
Jul 23 15:49:57.266900 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266733    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32ce7a51-fb39-4685-baa2-3ab6176d92d9-ca-trust-extracted\") pod \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\" (UID: \"32ce7a51-fb39-4685-baa2-3ab6176d92d9\") "
Jul 23 15:49:57.266951 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266929    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "32ce7a51-fb39-4685-baa2-3ab6176d92d9" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:49:57.267019 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.266950    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "32ce7a51-fb39-4685-baa2-3ab6176d92d9" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:49:57.269027 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.269002    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "32ce7a51-fb39-4685-baa2-3ab6176d92d9" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:49:57.269138 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.269115    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "32ce7a51-fb39-4685-baa2-3ab6176d92d9" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:49:57.269195 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.269173    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-kube-api-access-25547" (OuterVolumeSpecName: "kube-api-access-25547") pod "32ce7a51-fb39-4685-baa2-3ab6176d92d9" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9"). InnerVolumeSpecName "kube-api-access-25547". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:49:57.269294 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.269274    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "32ce7a51-fb39-4685-baa2-3ab6176d92d9" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:49:57.269382 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.269362    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "32ce7a51-fb39-4685-baa2-3ab6176d92d9" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:49:57.275445 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.275425    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce7a51-fb39-4685-baa2-3ab6176d92d9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "32ce7a51-fb39-4685-baa2-3ab6176d92d9" (UID: "32ce7a51-fb39-4685-baa2-3ab6176d92d9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:49:57.367389 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.367369    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-tls\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:49:57.367389 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.367387    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25547\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-kube-api-access-25547\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:49:57.367507 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.367397    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-image-registry-private-configuration\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:49:57.367507 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.367407    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32ce7a51-fb39-4685-baa2-3ab6176d92d9-ca-trust-extracted\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:49:57.367507 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.367415    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-registry-certificates\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:49:57.367507 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.367423    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32ce7a51-fb39-4685-baa2-3ab6176d92d9-installation-pull-secrets\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:49:57.367507 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.367430    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32ce7a51-fb39-4685-baa2-3ab6176d92d9-trusted-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:49:57.367507 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.367439    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32ce7a51-fb39-4685-baa2-3ab6176d92d9-bound-sa-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:49:57.539624 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.539599    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"]
Jul 23 15:49:57.543921 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:57.543897    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69c8dd7b7c-dmdrf"]
Jul 23 15:49:58.808511 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:49:58.808479    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ce7a51-fb39-4685-baa2-3ab6176d92d9" path="/var/lib/kubelet/pods/32ce7a51-fb39-4685-baa2-3ab6176d92d9/volumes"
Jul 23 15:50:06.474148 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.474115    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kbp86"]
Jul 23 15:50:06.474573 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.474389    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ce7a51-fb39-4685-baa2-3ab6176d92d9" containerName="registry"
Jul 23 15:50:06.474573 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.474403    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ce7a51-fb39-4685-baa2-3ab6176d92d9" containerName="registry"
Jul 23 15:50:06.474573 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.474455    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ce7a51-fb39-4685-baa2-3ab6176d92d9" containerName="registry"
Jul 23 15:50:06.514085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.514051    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.517561 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.517536    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 15:50:06.517736 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.517720    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 15:50:06.517822 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.517742    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9nsfz\""
Jul 23 15:50:06.517822 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.517792    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 15:50:06.518466 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.518447    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 15:50:06.518546 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.518496    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 15:50:06.518792 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.518777    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 15:50:06.529028 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529003    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-tls\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.529122 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529041    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.529122 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529065    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-root\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.529195 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529127    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-sys\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.529195 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-accelerators-collector-config\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.529262 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-textfile\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.529262 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529246    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/404a862c-f038-4943-b5ec-d94dd6cc0282-metrics-client-ca\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.529332 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529312    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-wtmp\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.529374 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.529356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krdjb\" (UniqueName: \"kubernetes.io/projected/404a862c-f038-4943-b5ec-d94dd6cc0282-kube-api-access-krdjb\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.629681 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-textfile\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.629836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629692    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/404a862c-f038-4943-b5ec-d94dd6cc0282-metrics-client-ca\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.629836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629729    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-wtmp\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.629836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629768    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdjb\" (UniqueName: \"kubernetes.io/projected/404a862c-f038-4943-b5ec-d94dd6cc0282-kube-api-access-krdjb\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.629836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-tls\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.629836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629852    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-root\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629881    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-sys\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629908    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-accelerators-collector-config\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:50:06.629939    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.629954    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-wtmp\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:50:06.630012    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-tls podName:404a862c-f038-4943-b5ec-d94dd6cc0282 nodeName:}" failed. No retries permitted until 2026-07-23 15:50:07.129991603 +0000 UTC m=+94.755526068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-tls") pod "node-exporter-kbp86" (UID: "404a862c-f038-4943-b5ec-d94dd6cc0282") : secret "node-exporter-tls" not found
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.630016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-root\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.630018    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-textfile\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.630080    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/404a862c-f038-4943-b5ec-d94dd6cc0282-sys\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630672 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.630653    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-accelerators-collector-config\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.630672 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.630662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/404a862c-f038-4943-b5ec-d94dd6cc0282-metrics-client-ca\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.632294 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.632275    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:06.639242 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:06.639220    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krdjb\" (UniqueName: \"kubernetes.io/projected/404a862c-f038-4943-b5ec-d94dd6cc0282-kube-api-access-krdjb\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:07.134945 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:07.134910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-tls\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:07.137032 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:07.137013    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/404a862c-f038-4943-b5ec-d94dd6cc0282-node-exporter-tls\") pod \"node-exporter-kbp86\" (UID: \"404a862c-f038-4943-b5ec-d94dd6cc0282\") " pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:07.424327 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:07.424251    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kbp86"
Jul 23 15:50:08.238047 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:08.238001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kbp86" event={"ID":"404a862c-f038-4943-b5ec-d94dd6cc0282","Type":"ContainerStarted","Data":"143e7d073873bffb3c85e1f5a3b476f3e564785602ccb7aadc8f5a8a9cc04455"}
Jul 23 15:50:09.242590 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:09.242556    2555 generic.go:356] "Generic (PLEG): container finished" podID="404a862c-f038-4943-b5ec-d94dd6cc0282" containerID="fc27ee727d53de101a0f2af434360eef0bd4506d9174e82c6ebf9d2a35891b89" exitCode=0
Jul 23 15:50:09.243066 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:09.242643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kbp86" event={"ID":"404a862c-f038-4943-b5ec-d94dd6cc0282","Type":"ContainerDied","Data":"fc27ee727d53de101a0f2af434360eef0bd4506d9174e82c6ebf9d2a35891b89"}
Jul 23 15:50:10.247990 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:10.247951    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kbp86" event={"ID":"404a862c-f038-4943-b5ec-d94dd6cc0282","Type":"ContainerStarted","Data":"c3a6416b7b833dddfcf4bf315e3a00ba98fdf0c1ed8ff896aa01b3d65939eadf"}
Jul 23 15:50:10.248326 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:10.248003    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kbp86" event={"ID":"404a862c-f038-4943-b5ec-d94dd6cc0282","Type":"ContainerStarted","Data":"b9b6b611227fdd81f822ee76dc1d35fbb6c72a8d6790d1e69911598ee69e1142"}
Jul 23 15:50:10.268894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:10.268850    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kbp86" podStartSLOduration=3.292822926 podStartE2EDuration="4.268835581s" totalImagesPullingTime="914.351695ms" totalInitContainerRuntime="61.66096ms" isStatefulPod=true podCreationTimestamp="2026-07-23 15:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:50:10.267054094 +0000 UTC m=+97.892588580" watchObservedRunningTime="2026-07-23 15:50:10.268835581 +0000 UTC m=+97.894370055"
Jul 23 15:50:18.173918 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:18.173888    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zr2d7"
Jul 23 15:50:32.157176 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:50:32.157140    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g4zf2_10e4692c-174e-4e57-a7b0-7ec577cea880/serve-healthcheck-canary/0.log"
Jul 23 15:53:32.695439 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:53:32.695408    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:53:32.699649 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:53:32.699625    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:53:32.701772 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:53:32.701743    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 15:54:45.747386 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.747304    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"]
Jul 23 15:54:45.750377 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.750360    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.755723 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.755698    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 15:54:45.755871 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.755726    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 15:54:45.756493 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.756314    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 15:54:45.757103 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.757083    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 15:54:45.765084 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.765066    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9w9v5\""
Jul 23 15:54:45.790003 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.789976    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"]
Jul 23 15:54:45.824006 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.823982    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b00d5738-76c3-4097-ac99-453517b7cc4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.824189 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.824167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b00d5738-76c3-4097-ac99-453517b7cc4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.824280 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.824206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vmg\" (UniqueName: \"kubernetes.io/projected/b00d5738-76c3-4097-ac99-453517b7cc4f-kube-api-access-k6vmg\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.824330 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.824291    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b00d5738-76c3-4097-ac99-453517b7cc4f-manifests\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.824366 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.824351    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b00d5738-76c3-4097-ac99-453517b7cc4f-tmp\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.924652 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.924626    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b00d5738-76c3-4097-ac99-453517b7cc4f-tmp\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.924744 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.924664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b00d5738-76c3-4097-ac99-453517b7cc4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.924744 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.924686    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b00d5738-76c3-4097-ac99-453517b7cc4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.924887 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.924869    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vmg\" (UniqueName: \"kubernetes.io/projected/b00d5738-76c3-4097-ac99-453517b7cc4f-kube-api-access-k6vmg\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.924956 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.924940    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b00d5738-76c3-4097-ac99-453517b7cc4f-manifests\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.925273 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.925254    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b00d5738-76c3-4097-ac99-453517b7cc4f-manifests\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.926855 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.926835    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b00d5738-76c3-4097-ac99-453517b7cc4f-tmp\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.927054 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.927038    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b00d5738-76c3-4097-ac99-453517b7cc4f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.927111 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.927097    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b00d5738-76c3-4097-ac99-453517b7cc4f-webhook-cert\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:45.934424 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:45.934397    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vmg\" (UniqueName: \"kubernetes.io/projected/b00d5738-76c3-4097-ac99-453517b7cc4f-kube-api-access-k6vmg\") pod \"opendatahub-operator-controller-manager-646b5d5c79-g8v4n\" (UID: \"b00d5738-76c3-4097-ac99-453517b7cc4f\") " pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:46.059913 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:46.059858    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:54:46.188156 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:46.188127    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"]
Jul 23 15:54:46.191152 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:54:46.191123    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00d5738_76c3_4097_ac99_453517b7cc4f.slice/crio-b154cb5583897ca8c900bfdd465d3c686814890b1df1dfe34792d0eaead98f60 WatchSource:0}: Error finding container b154cb5583897ca8c900bfdd465d3c686814890b1df1dfe34792d0eaead98f60: Status 404 returned error can't find the container with id b154cb5583897ca8c900bfdd465d3c686814890b1df1dfe34792d0eaead98f60
Jul 23 15:54:46.192845 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:46.192816    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 15:54:47.127431 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:47.127385    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n" event={"ID":"b00d5738-76c3-4097-ac99-453517b7cc4f","Type":"ContainerStarted","Data":"b154cb5583897ca8c900bfdd465d3c686814890b1df1dfe34792d0eaead98f60"}
Jul 23 15:54:50.140216 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:50.140180    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n" event={"ID":"b00d5738-76c3-4097-ac99-453517b7cc4f","Type":"ContainerStarted","Data":"eca2c7ffb40cb5a88be5c04aed60465ce080f91ee59168e6722b33c11e433f4f"}
Jul 23 15:54:51.144518 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:51.144484    2555 generic.go:356] "Generic (PLEG): container finished" podID="b00d5738-76c3-4097-ac99-453517b7cc4f" containerID="eca2c7ffb40cb5a88be5c04aed60465ce080f91ee59168e6722b33c11e433f4f" exitCode=0
Jul 23 15:54:51.144936 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:51.144572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n" event={"ID":"b00d5738-76c3-4097-ac99-453517b7cc4f","Type":"ContainerDied","Data":"eca2c7ffb40cb5a88be5c04aed60465ce080f91ee59168e6722b33c11e433f4f"}
Jul 23 15:54:52.150327 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:52.150289    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n" event={"ID":"b00d5738-76c3-4097-ac99-453517b7cc4f","Type":"ContainerStarted","Data":"9fe64522f959db37f66e89d96de0e1334afd356074e97a33036eea6c6bd7cf20"}
Jul 23 15:54:52.150800 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:54:52.150377    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:55:03.155365 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:03.155334    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n"
Jul 23 15:55:03.201678 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:03.201629    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-646b5d5c79-g8v4n" podStartSLOduration=13.338940551 podStartE2EDuration="18.201612934s" totalImagesPullingTime="3.829574561s" totalInitContainerRuntime="1.033097822s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:54:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:54:52.236098035 +0000 UTC m=+379.861632510" watchObservedRunningTime="2026-07-23 15:55:03.201612934 +0000 UTC m=+390.827147474"
Jul 23 15:55:06.198382 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.198346    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"]
Jul 23 15:55:06.206160 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.206134    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.212034 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.212004    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 23 15:55:06.212187 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.212100    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 23 15:55:06.212260 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.212181    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k8sm9\""
Jul 23 15:55:06.212260 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.212247    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 15:55:06.212260 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.212252    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 15:55:06.235336 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.235309    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"]
Jul 23 15:55:06.263823 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.263782    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-tmp\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.263999 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.263841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-tls-certs\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.263999 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.263967    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz69s\" (UniqueName: \"kubernetes.io/projected/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-kube-api-access-vz69s\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.365059 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.365028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz69s\" (UniqueName: \"kubernetes.io/projected/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-kube-api-access-vz69s\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.365235 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.365066    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-tmp\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.365235 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.365101    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-tls-certs\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.367464 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.367427    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-tmp\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.367616 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.367601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-tls-certs\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.396228 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.396205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz69s\" (UniqueName: \"kubernetes.io/projected/b4d21f9f-fc16-4d61-b17f-0c020e4112d6-kube-api-access-vz69s\") pod \"kube-auth-proxy-5fb5dcd56-zfn4h\" (UID: \"b4d21f9f-fc16-4d61-b17f-0c020e4112d6\") " pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.487016 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.486938    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xkgwl"]
Jul 23 15:55:06.489979 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.489962    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:06.492430 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.492409    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xmzk6\""
Jul 23 15:55:06.493147 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.493133    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 23 15:55:06.514847 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.514824    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"
Jul 23 15:55:06.567665 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.567635    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/819a959a-c089-46a6-b0ed-d326f900f1c4-cert\") pod \"odh-model-controller-7998c67dc4-xkgwl\" (UID: \"819a959a-c089-46a6-b0ed-d326f900f1c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:06.567825 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.567686    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94xw\" (UniqueName: \"kubernetes.io/projected/819a959a-c089-46a6-b0ed-d326f900f1c4-kube-api-access-z94xw\") pod \"odh-model-controller-7998c67dc4-xkgwl\" (UID: \"819a959a-c089-46a6-b0ed-d326f900f1c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:06.642392 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.642359    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xkgwl"]
Jul 23 15:55:06.667464 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.667436    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h"]
Jul 23 15:55:06.668211 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.668191    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/819a959a-c089-46a6-b0ed-d326f900f1c4-cert\") pod \"odh-model-controller-7998c67dc4-xkgwl\" (UID: \"819a959a-c089-46a6-b0ed-d326f900f1c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:06.668299 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.668238    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94xw\" (UniqueName: \"kubernetes.io/projected/819a959a-c089-46a6-b0ed-d326f900f1c4-kube-api-access-z94xw\") pod \"odh-model-controller-7998c67dc4-xkgwl\" (UID: \"819a959a-c089-46a6-b0ed-d326f900f1c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:06.668299 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:55:06.668266    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d21f9f_fc16_4d61_b17f_0c020e4112d6.slice/crio-34e6ff0e637b88ffead3fb14f7e25decacdb7420e50921a157b0d4144cdaf52a WatchSource:0}: Error finding container 34e6ff0e637b88ffead3fb14f7e25decacdb7420e50921a157b0d4144cdaf52a: Status 404 returned error can't find the container with id 34e6ff0e637b88ffead3fb14f7e25decacdb7420e50921a157b0d4144cdaf52a
Jul 23 15:55:06.668382 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:55:06.668347    2555 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 23 15:55:06.668432 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:55:06.668420    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/819a959a-c089-46a6-b0ed-d326f900f1c4-cert podName:819a959a-c089-46a6-b0ed-d326f900f1c4 nodeName:}" failed. No retries permitted until 2026-07-23 15:55:07.168398881 +0000 UTC m=+394.793933355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/819a959a-c089-46a6-b0ed-d326f900f1c4-cert") pod "odh-model-controller-7998c67dc4-xkgwl" (UID: "819a959a-c089-46a6-b0ed-d326f900f1c4") : secret "odh-model-controller-webhook-cert" not found
Jul 23 15:55:06.707478 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:06.707446    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94xw\" (UniqueName: \"kubernetes.io/projected/819a959a-c089-46a6-b0ed-d326f900f1c4-kube-api-access-z94xw\") pod \"odh-model-controller-7998c67dc4-xkgwl\" (UID: \"819a959a-c089-46a6-b0ed-d326f900f1c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:07.172561 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:07.172523    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/819a959a-c089-46a6-b0ed-d326f900f1c4-cert\") pod \"odh-model-controller-7998c67dc4-xkgwl\" (UID: \"819a959a-c089-46a6-b0ed-d326f900f1c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:07.174876 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:07.174855    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/819a959a-c089-46a6-b0ed-d326f900f1c4-cert\") pod \"odh-model-controller-7998c67dc4-xkgwl\" (UID: \"819a959a-c089-46a6-b0ed-d326f900f1c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:07.203185 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:07.203153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h" event={"ID":"b4d21f9f-fc16-4d61-b17f-0c020e4112d6","Type":"ContainerStarted","Data":"34e6ff0e637b88ffead3fb14f7e25decacdb7420e50921a157b0d4144cdaf52a"}
Jul 23 15:55:07.399840 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:07.399807    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:07.592496 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:07.592472    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xkgwl"]
Jul 23 15:55:07.593218 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:55:07.593191    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819a959a_c089_46a6_b0ed_d326f900f1c4.slice/crio-786c324f0c2dff83870d3a5717ba2d0ee6ea8fdcb7d74efa069b8e9e38e10edd WatchSource:0}: Error finding container 786c324f0c2dff83870d3a5717ba2d0ee6ea8fdcb7d74efa069b8e9e38e10edd: Status 404 returned error can't find the container with id 786c324f0c2dff83870d3a5717ba2d0ee6ea8fdcb7d74efa069b8e9e38e10edd
Jul 23 15:55:08.023213 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.023178    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mcdvg"]
Jul 23 15:55:08.026248 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.026230    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:08.030606 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.030584    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 23 15:55:08.031208 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.031189    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-s2cgs\""
Jul 23 15:55:08.050788 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.050762    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mcdvg"]
Jul 23 15:55:08.080436 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.080397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9ld\" (UniqueName: \"kubernetes.io/projected/85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c-kube-api-access-vx9ld\") pod \"model-serving-api-54d89bd5fc-mcdvg\" (UID: \"85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:08.080612 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.080568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c-tls-certs\") pod \"model-serving-api-54d89bd5fc-mcdvg\" (UID: \"85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:08.182053 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.182017    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c-tls-certs\") pod \"model-serving-api-54d89bd5fc-mcdvg\" (UID: \"85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:08.182220 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.182065    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9ld\" (UniqueName: \"kubernetes.io/projected/85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c-kube-api-access-vx9ld\") pod \"model-serving-api-54d89bd5fc-mcdvg\" (UID: \"85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:08.184472 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.184444    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c-tls-certs\") pod \"model-serving-api-54d89bd5fc-mcdvg\" (UID: \"85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:08.212912 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.212870    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl" event={"ID":"819a959a-c089-46a6-b0ed-d326f900f1c4","Type":"ContainerStarted","Data":"786c324f0c2dff83870d3a5717ba2d0ee6ea8fdcb7d74efa069b8e9e38e10edd"}
Jul 23 15:55:08.228466 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.228436    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9ld\" (UniqueName: \"kubernetes.io/projected/85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c-kube-api-access-vx9ld\") pod \"model-serving-api-54d89bd5fc-mcdvg\" (UID: \"85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:08.336414 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.336328    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:08.519124 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.519041    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mcdvg"]
Jul 23 15:55:08.526796 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:55:08.526730    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e86b8f_e985_4d3f_b194_8f0d3c6e3a2c.slice/crio-830fe799c9cbc4edd3a5c927f15a7fb56da148d697007bfae036e978e422b489 WatchSource:0}: Error finding container 830fe799c9cbc4edd3a5c927f15a7fb56da148d697007bfae036e978e422b489: Status 404 returned error can't find the container with id 830fe799c9cbc4edd3a5c927f15a7fb56da148d697007bfae036e978e422b489
Jul 23 15:55:08.922822 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.922782    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"]
Jul 23 15:55:08.926406 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.926381    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:08.929699 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.929677    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 15:55:08.930042 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.930021    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 23 15:55:08.930261 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.930242    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 23 15:55:08.930365 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.930261    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 23 15:55:08.930431 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.930364    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ks8jw\""
Jul 23 15:55:08.945139 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.945110    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"]
Jul 23 15:55:08.988078 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.988043    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-local-certs\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:08.988272 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.988090    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:08.988272 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.988128    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmv6\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-kube-api-access-xhmv6\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:08.988272 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.988194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-cacerts\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:08.988272 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.988251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:08.988490 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.988298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-token\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:08.988490 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:08.988324    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.089739 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.089702    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-cacerts\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.089931 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.089798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.089931 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.089847    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-token\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.089931 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.089875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.089931 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.089901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-local-certs\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.089931 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.089923    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.090144 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.089954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmv6\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-kube-api-access-xhmv6\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.090467 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.090440    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.093057 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.092929    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.093057 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.093022    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-cacerts\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.093225 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.093129    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-local-certs\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.093225 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.093207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.107460 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.107431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-token\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.108258 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.108221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmv6\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-kube-api-access-xhmv6\") pod \"istiod-openshift-gateway-7574459656-cgzxp\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:09.218947 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.218862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg" event={"ID":"85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c","Type":"ContainerStarted","Data":"830fe799c9cbc4edd3a5c927f15a7fb56da148d697007bfae036e978e422b489"}
Jul 23 15:55:09.239112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:09.239081    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:11.508515 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:11.508326    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"]
Jul 23 15:55:11.511145 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:55:11.511119    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795f15a5_51b3_42f8_a9f2_5c545a5aaa45.slice/crio-2d5b4b3c0fd0c8f94032a212519462b84dc7b9416288d5aaa6efaa39164d6aee WatchSource:0}: Error finding container 2d5b4b3c0fd0c8f94032a212519462b84dc7b9416288d5aaa6efaa39164d6aee: Status 404 returned error can't find the container with id 2d5b4b3c0fd0c8f94032a212519462b84dc7b9416288d5aaa6efaa39164d6aee
Jul 23 15:55:12.231367 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.231320    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl" event={"ID":"819a959a-c089-46a6-b0ed-d326f900f1c4","Type":"ContainerStarted","Data":"524b2ce893cd9037ca7b242f3c0f9b528129e459ab5707f3b934fa79dcc979f6"}
Jul 23 15:55:12.231559 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.231436    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:12.232857 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.232829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp" event={"ID":"795f15a5-51b3-42f8-a9f2-5c545a5aaa45","Type":"ContainerStarted","Data":"2d5b4b3c0fd0c8f94032a212519462b84dc7b9416288d5aaa6efaa39164d6aee"}
Jul 23 15:55:12.234377 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.234341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h" event={"ID":"b4d21f9f-fc16-4d61-b17f-0c020e4112d6","Type":"ContainerStarted","Data":"2b1e409ba403d48f1961d9ed2da2dac3f4986b802b17558aaf68f56878fc75a7"}
Jul 23 15:55:12.236048 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.236019    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg" event={"ID":"85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c","Type":"ContainerStarted","Data":"450fcd7740cc9bb29cb5db9ddb71910f9f8fb6320cc0f61bce7b5693bc9ab675"}
Jul 23 15:55:12.236160 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.236142    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:12.249470 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.249428    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl" podStartSLOduration=2.488155162 podStartE2EDuration="6.249414457s" totalImagesPullingTime="3.761259295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:55:12.248560756 +0000 UTC m=+399.874095233" watchObservedRunningTime="2026-07-23 15:55:12.249414457 +0000 UTC m=+399.874948933"
Jul 23 15:55:12.271622 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.271578    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fb5dcd56-zfn4h" podStartSLOduration=1.57953359 podStartE2EDuration="6.271564085s" totalImagesPullingTime="4.692030495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:55:12.270426646 +0000 UTC m=+399.895961124" watchObservedRunningTime="2026-07-23 15:55:12.271564085 +0000 UTC m=+399.897098561"
Jul 23 15:55:12.295669 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:12.295616    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg" podStartSLOduration=2.425559486 podStartE2EDuration="5.295601904s" totalImagesPullingTime="2.870042418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:55:12.293507497 +0000 UTC m=+399.919041974" watchObservedRunningTime="2026-07-23 15:55:12.295601904 +0000 UTC m=+399.921136380"
Jul 23 15:55:14.417712 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:14.417670    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:55:14.418037 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:14.417771    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:55:15.249893 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:15.249858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp" event={"ID":"795f15a5-51b3-42f8-a9f2-5c545a5aaa45","Type":"ContainerStarted","Data":"2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2"}
Jul 23 15:55:15.250135 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:15.250104    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:15.251701 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:15.251675    2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-cgzxp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 15:55:15.251821 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:15.251725    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp" podUID="795f15a5-51b3-42f8-a9f2-5c545a5aaa45" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 15:55:15.307372 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:15.307323    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp" podStartSLOduration=4.403250625 podStartE2EDuration="7.307310491s" totalImagesPullingTime="2.904059866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:55:15.305833574 +0000 UTC m=+402.931368050" watchObservedRunningTime="2026-07-23 15:55:15.307310491 +0000 UTC m=+402.932844967"
Jul 23 15:55:16.254244 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.254204    2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-cgzxp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 15:55:16.254604 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.254266    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp" podUID="795f15a5-51b3-42f8-a9f2-5c545a5aaa45" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 15:55:16.903866 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.903827    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"]
Jul 23 15:55:16.906613 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.906581    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.910669 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.910641    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k7fs5\""
Jul 23 15:55:16.925825 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.925797    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"]
Jul 23 15:55:16.956126 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.956267 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956142    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.956267 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp7kb\" (UniqueName: \"kubernetes.io/projected/50ba0115-92ad-427f-91a6-4108fb2bda5f-kube-api-access-tp7kb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.956371 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.956371 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956333    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ba0115-92ad-427f-91a6-4108fb2bda5f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.956467 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956367    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.956467 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956391    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.956467 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:16.956467 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:16.956449    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.056852 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.056824    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.056977 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.056857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.056977 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.056880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.056977 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.056904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.056977 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.056962    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057196 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.056991    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057196 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.057022    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7kb\" (UniqueName: \"kubernetes.io/projected/50ba0115-92ad-427f-91a6-4108fb2bda5f-kube-api-access-tp7kb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057196 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.057059    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057196 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.057102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ba0115-92ad-427f-91a6-4108fb2bda5f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057427 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.057319    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057498 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.057474    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057559 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.057495    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057619 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.057597    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.057662 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.057648    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50ba0115-92ad-427f-91a6-4108fb2bda5f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.059286 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.059264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.059521 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.059500    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.080933 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.080911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50ba0115-92ad-427f-91a6-4108fb2bda5f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.098744 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.098714    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7kb\" (UniqueName: \"kubernetes.io/projected/50ba0115-92ad-427f-91a6-4108fb2bda5f-kube-api-access-tp7kb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x66t8\" (UID: \"50ba0115-92ad-427f-91a6-4108fb2bda5f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.216563 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.216500    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:17.365406 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:17.365377    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"]
Jul 23 15:55:17.368303 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:55:17.368275    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ba0115_92ad_427f_91a6_4108fb2bda5f.slice/crio-76621f152e1b6a9d660c6394d16219926568df2f3feffadc0afb1e2f3680741a WatchSource:0}: Error finding container 76621f152e1b6a9d660c6394d16219926568df2f3feffadc0afb1e2f3680741a: Status 404 returned error can't find the container with id 76621f152e1b6a9d660c6394d16219926568df2f3feffadc0afb1e2f3680741a
Jul 23 15:55:18.264297 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.264256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8" event={"ID":"50ba0115-92ad-427f-91a6-4108fb2bda5f","Type":"ContainerStarted","Data":"76621f152e1b6a9d660c6394d16219926568df2f3feffadc0afb1e2f3680741a"}
Jul 23 15:55:18.510798 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.510762    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"]
Jul 23 15:55:18.514203 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.514176    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:18.522067 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.522008    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 23 15:55:18.522189 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.522012    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-z4w7b\""
Jul 23 15:55:18.544209 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.544100    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"]
Jul 23 15:55:18.545530 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.545502    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"]
Jul 23 15:55:18.549022 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.549003    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:18.552632 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.552610    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 23 15:55:18.566686 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.566663    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-b8srd\""
Jul 23 15:55:18.568605 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.568562    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3312d672-b038-4e98-a46e-e1bbbe6f71be-cert\") pod \"kserve-controller-manager-d6f4b65b8-rr68k\" (UID: \"3312d672-b038-4e98-a46e-e1bbbe6f71be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:18.568740 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.568650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7h9l\" (UniqueName: \"kubernetes.io/projected/3312d672-b038-4e98-a46e-e1bbbe6f71be-kube-api-access-n7h9l\") pod \"kserve-controller-manager-d6f4b65b8-rr68k\" (UID: \"3312d672-b038-4e98-a46e-e1bbbe6f71be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:18.609836 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.609799    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"]
Jul 23 15:55:18.670056 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.670020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7h9l\" (UniqueName: \"kubernetes.io/projected/3312d672-b038-4e98-a46e-e1bbbe6f71be-kube-api-access-n7h9l\") pod \"kserve-controller-manager-d6f4b65b8-rr68k\" (UID: \"3312d672-b038-4e98-a46e-e1bbbe6f71be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:18.670233 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.670196    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5jrwb\" (UID: \"1689b2c4-e09a-4d76-b7ac-a0543a62eedd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:18.670307 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.670263    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3312d672-b038-4e98-a46e-e1bbbe6f71be-cert\") pod \"kserve-controller-manager-d6f4b65b8-rr68k\" (UID: \"3312d672-b038-4e98-a46e-e1bbbe6f71be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:18.670307 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.670294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmpb\" (UniqueName: \"kubernetes.io/projected/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-kube-api-access-dpmpb\") pod \"llmisvc-controller-manager-777c6bd7d5-5jrwb\" (UID: \"1689b2c4-e09a-4d76-b7ac-a0543a62eedd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:18.670438 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:55:18.670417    2555 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 23 15:55:18.670508 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:55:18.670487    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3312d672-b038-4e98-a46e-e1bbbe6f71be-cert podName:3312d672-b038-4e98-a46e-e1bbbe6f71be nodeName:}" failed. No retries permitted until 2026-07-23 15:55:19.170465991 +0000 UTC m=+406.796000459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3312d672-b038-4e98-a46e-e1bbbe6f71be-cert") pod "kserve-controller-manager-d6f4b65b8-rr68k" (UID: "3312d672-b038-4e98-a46e-e1bbbe6f71be") : secret "kserve-webhook-server-cert" not found
Jul 23 15:55:18.695653 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.695619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7h9l\" (UniqueName: \"kubernetes.io/projected/3312d672-b038-4e98-a46e-e1bbbe6f71be-kube-api-access-n7h9l\") pod \"kserve-controller-manager-d6f4b65b8-rr68k\" (UID: \"3312d672-b038-4e98-a46e-e1bbbe6f71be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:18.771671 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.771630    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5jrwb\" (UID: \"1689b2c4-e09a-4d76-b7ac-a0543a62eedd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:18.771878 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:55:18.771760    2555 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 23 15:55:18.771878 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:55:18.771837    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-cert podName:1689b2c4-e09a-4d76-b7ac-a0543a62eedd nodeName:}" failed. No retries permitted until 2026-07-23 15:55:19.271818153 +0000 UTC m=+406.897352628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-cert") pod "llmisvc-controller-manager-777c6bd7d5-5jrwb" (UID: "1689b2c4-e09a-4d76-b7ac-a0543a62eedd") : secret "llmisvc-webhook-server-cert" not found
Jul 23 15:55:18.772011 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.771764    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmpb\" (UniqueName: \"kubernetes.io/projected/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-kube-api-access-dpmpb\") pod \"llmisvc-controller-manager-777c6bd7d5-5jrwb\" (UID: \"1689b2c4-e09a-4d76-b7ac-a0543a62eedd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:18.807168 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:18.807064    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmpb\" (UniqueName: \"kubernetes.io/projected/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-kube-api-access-dpmpb\") pod \"llmisvc-controller-manager-777c6bd7d5-5jrwb\" (UID: \"1689b2c4-e09a-4d76-b7ac-a0543a62eedd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:19.176993 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:19.176951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3312d672-b038-4e98-a46e-e1bbbe6f71be-cert\") pod \"kserve-controller-manager-d6f4b65b8-rr68k\" (UID: \"3312d672-b038-4e98-a46e-e1bbbe6f71be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:19.179595 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:19.179561    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3312d672-b038-4e98-a46e-e1bbbe6f71be-cert\") pod \"kserve-controller-manager-d6f4b65b8-rr68k\" (UID: \"3312d672-b038-4e98-a46e-e1bbbe6f71be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:19.255523 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:19.255490    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:19.277903 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:19.277872    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5jrwb\" (UID: \"1689b2c4-e09a-4d76-b7ac-a0543a62eedd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:19.280924 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:19.280900    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1689b2c4-e09a-4d76-b7ac-a0543a62eedd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5jrwb\" (UID: \"1689b2c4-e09a-4d76-b7ac-a0543a62eedd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:19.427425 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:19.427304    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:19.463012 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:19.462984    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:20.248251 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:20.248215    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:55:20.248555 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:20.248300    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:55:20.248555 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:20.248343    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:55:20.345161 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:55:20.345117    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3312d672_b038_4e98_a46e_e1bbbe6f71be.slice/crio-35f7a601653115d902c98de7aea0a5d3b75c6dd0951c99e3067ff3295c960185 WatchSource:0}: Error finding container 35f7a601653115d902c98de7aea0a5d3b75c6dd0951c99e3067ff3295c960185: Status 404 returned error can't find the container with id 35f7a601653115d902c98de7aea0a5d3b75c6dd0951c99e3067ff3295c960185
Jul 23 15:55:20.349002 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:20.348973    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"]
Jul 23 15:55:20.350924 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:20.350891    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"]
Jul 23 15:55:20.352583 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:55:20.352553    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1689b2c4_e09a_4d76_b7ac_a0543a62eedd.slice/crio-c58eb9862e4ae7b8b39003c433615d100c439a976175cc0f845f986f0006f9d6 WatchSource:0}: Error finding container c58eb9862e4ae7b8b39003c433615d100c439a976175cc0f845f986f0006f9d6: Status 404 returned error can't find the container with id c58eb9862e4ae7b8b39003c433615d100c439a976175cc0f845f986f0006f9d6
Jul 23 15:55:21.286817 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:21.286783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k" event={"ID":"3312d672-b038-4e98-a46e-e1bbbe6f71be","Type":"ContainerStarted","Data":"35f7a601653115d902c98de7aea0a5d3b75c6dd0951c99e3067ff3295c960185"}
Jul 23 15:55:21.288798 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:21.288740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb" event={"ID":"1689b2c4-e09a-4d76-b7ac-a0543a62eedd","Type":"ContainerStarted","Data":"c58eb9862e4ae7b8b39003c433615d100c439a976175cc0f845f986f0006f9d6"}
Jul 23 15:55:21.290530 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:21.290502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8" event={"ID":"50ba0115-92ad-427f-91a6-4108fb2bda5f","Type":"ContainerStarted","Data":"f2dfeb191d880b68d02addaf9a9712350182fd05084dd7f7ff15521ed0cbb234"}
Jul 23 15:55:21.319798 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:21.319707    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8" podStartSLOduration=2.442147244 podStartE2EDuration="5.319684413s" totalImagesPullingTime="2.877537169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:55:21.316959748 +0000 UTC m=+408.942494238" watchObservedRunningTime="2026-07-23 15:55:21.319684413 +0000 UTC m=+408.945218892"
Jul 23 15:55:21.604690 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:21.604608    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"]
Jul 23 15:55:21.604896 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:21.604868    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp" podUID="795f15a5-51b3-42f8-a9f2-5c545a5aaa45" containerName="discovery" containerID="cri-o://2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2" gracePeriod=30
Jul 23 15:55:21.949995 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:21.949967    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:22.001729 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.001694    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-kubeconfig\") pod \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") "
Jul 23 15:55:22.001928 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.001738    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-cacerts\") pod \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") "
Jul 23 15:55:22.001928 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.001817    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-local-certs\") pod \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") "
Jul 23 15:55:22.001928 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.001878    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-dns-cert\") pod \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") "
Jul 23 15:55:22.001928 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.001916    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-token\") pod \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") "
Jul 23 15:55:22.002136 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.001955    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhmv6\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-kube-api-access-xhmv6\") pod \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") "
Jul 23 15:55:22.002136 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.002004    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-ca-configmap\") pod \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\" (UID: \"795f15a5-51b3-42f8-a9f2-5c545a5aaa45\") "
Jul 23 15:55:22.003001 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.002970    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "795f15a5-51b3-42f8-a9f2-5c545a5aaa45" (UID: "795f15a5-51b3-42f8-a9f2-5c545a5aaa45"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 15:55:22.005394 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.005353    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-local-certs" (OuterVolumeSpecName: "local-certs") pod "795f15a5-51b3-42f8-a9f2-5c545a5aaa45" (UID: "795f15a5-51b3-42f8-a9f2-5c545a5aaa45"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:55:22.005533 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.005487    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-token" (OuterVolumeSpecName: "istio-token") pod "795f15a5-51b3-42f8-a9f2-5c545a5aaa45" (UID: "795f15a5-51b3-42f8-a9f2-5c545a5aaa45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:55:22.005618 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.005586    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "795f15a5-51b3-42f8-a9f2-5c545a5aaa45" (UID: "795f15a5-51b3-42f8-a9f2-5c545a5aaa45"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:55:22.005812 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.005781    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "795f15a5-51b3-42f8-a9f2-5c545a5aaa45" (UID: "795f15a5-51b3-42f8-a9f2-5c545a5aaa45"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:55:22.007911 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.007875    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-cacerts" (OuterVolumeSpecName: "cacerts") pod "795f15a5-51b3-42f8-a9f2-5c545a5aaa45" (UID: "795f15a5-51b3-42f8-a9f2-5c545a5aaa45"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:55:22.007911 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.007887    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-kube-api-access-xhmv6" (OuterVolumeSpecName: "kube-api-access-xhmv6") pod "795f15a5-51b3-42f8-a9f2-5c545a5aaa45" (UID: "795f15a5-51b3-42f8-a9f2-5c545a5aaa45"). InnerVolumeSpecName "kube-api-access-xhmv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:55:22.103653 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.103619    2555 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-kubeconfig\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:55:22.103653 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.103649    2555 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-cacerts\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:55:22.103653 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.103660    2555 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-local-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:55:22.103903 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.103669    2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-dns-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:55:22.103903 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.103678    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:55:22.103903 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.103686    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhmv6\" (UniqueName: \"kubernetes.io/projected/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-kube-api-access-xhmv6\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:55:22.103903 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.103694    2555 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/795f15a5-51b3-42f8-a9f2-5c545a5aaa45-istio-csr-ca-configmap\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:55:22.216767 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.216719    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:22.221583 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.221562    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:22.295629 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.295593    2555 generic.go:356] "Generic (PLEG): container finished" podID="795f15a5-51b3-42f8-a9f2-5c545a5aaa45" containerID="2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2" exitCode=0
Jul 23 15:55:22.296110 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.295644    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"
Jul 23 15:55:22.296110 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.295674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp" event={"ID":"795f15a5-51b3-42f8-a9f2-5c545a5aaa45","Type":"ContainerDied","Data":"2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2"}
Jul 23 15:55:22.296110 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.295720    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp" event={"ID":"795f15a5-51b3-42f8-a9f2-5c545a5aaa45","Type":"ContainerDied","Data":"2d5b4b3c0fd0c8f94032a212519462b84dc7b9416288d5aaa6efaa39164d6aee"}
Jul 23 15:55:22.296110 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.295741    2555 scope.go:117] "RemoveContainer" containerID="2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2"
Jul 23 15:55:22.296327 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.296122    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:22.297199 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.297182    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x66t8"
Jul 23 15:55:22.304076 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.304059    2555 scope.go:117] "RemoveContainer" containerID="2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2"
Jul 23 15:55:22.304331 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:55:22.304311    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2\": container with ID starting with 2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2 not found: ID does not exist" containerID="2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2"
Jul 23 15:55:22.304384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.304339    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2"} err="failed to get container status \"2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2\": rpc error: code = NotFound desc = could not find container \"2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2\": container with ID starting with 2952c635241ba911c70e8374713528445fa2811c423e00396cd08f70c54eb6f2 not found: ID does not exist"
Jul 23 15:55:22.366384 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.366356    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"]
Jul 23 15:55:22.380302 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.380272    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-cgzxp"]
Jul 23 15:55:22.810459 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:22.810425    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795f15a5-51b3-42f8-a9f2-5c545a5aaa45" path="/var/lib/kubelet/pods/795f15a5-51b3-42f8-a9f2-5c545a5aaa45/volumes"
Jul 23 15:55:23.242553 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:23.242514    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-xkgwl"
Jul 23 15:55:23.244779 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:23.244737    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mcdvg"
Jul 23 15:55:24.307841 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:24.307802    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb" event={"ID":"1689b2c4-e09a-4d76-b7ac-a0543a62eedd","Type":"ContainerStarted","Data":"8cafcb8af1579cdcffda4ef5f20ae921b1fe0f9be7246f55610be933083ce557"}
Jul 23 15:55:24.308313 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:24.307974    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:24.309443 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:24.309416    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k" event={"ID":"3312d672-b038-4e98-a46e-e1bbbe6f71be","Type":"ContainerStarted","Data":"251927f15851cb0cc6366a3c0a69e5f8a0bd445e9ae50e1b02fb3b3f486883ce"}
Jul 23 15:55:24.309562 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:24.309552    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:24.338276 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:24.338222    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb" podStartSLOduration=2.940017252 podStartE2EDuration="6.338206656s" totalImagesPullingTime="3.398189404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:55:24.334226502 +0000 UTC m=+411.959760977" watchObservedRunningTime="2026-07-23 15:55:24.338206656 +0000 UTC m=+411.963741170"
Jul 23 15:55:55.314998 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:55.314906    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5jrwb"
Jul 23 15:55:55.318142 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:55.318122    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k"
Jul 23 15:55:55.334014 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:55:55.333972    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rr68k" podStartSLOduration=33.933712795 podStartE2EDuration="37.333956763s" totalImagesPullingTime="3.400243968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:55:24.36805692 +0000 UTC m=+411.993591395" watchObservedRunningTime="2026-07-23 15:55:55.333956763 +0000 UTC m=+442.959491238"
Jul 23 15:56:06.989313 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:06.989276    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p4nqt"]
Jul 23 15:56:06.989687 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:06.989564    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795f15a5-51b3-42f8-a9f2-5c545a5aaa45" containerName="discovery"
Jul 23 15:56:06.989687 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:06.989574    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="795f15a5-51b3-42f8-a9f2-5c545a5aaa45" containerName="discovery"
Jul 23 15:56:06.989687 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:06.989623    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="795f15a5-51b3-42f8-a9f2-5c545a5aaa45" containerName="discovery"
Jul 23 15:56:06.993215 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:06.993192    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt"
Jul 23 15:56:06.996324 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:06.996296    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-876bw\""
Jul 23 15:56:06.996431 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:06.996296    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 15:56:06.997145 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:06.997128    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 15:56:07.009483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.009458    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p4nqt"]
Jul 23 15:56:07.053940 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.053903    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mfwk\" (UniqueName: \"kubernetes.io/projected/9fbf1344-5b40-4019-ad13-0eacbc6469e2-kube-api-access-4mfwk\") pod \"kuadrant-operator-catalog-p4nqt\" (UID: \"9fbf1344-5b40-4019-ad13-0eacbc6469e2\") " pod="kuadrant-system/kuadrant-operator-catalog-p4nqt"
Jul 23 15:56:07.155066 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.155026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mfwk\" (UniqueName: \"kubernetes.io/projected/9fbf1344-5b40-4019-ad13-0eacbc6469e2-kube-api-access-4mfwk\") pod \"kuadrant-operator-catalog-p4nqt\" (UID: \"9fbf1344-5b40-4019-ad13-0eacbc6469e2\") " pod="kuadrant-system/kuadrant-operator-catalog-p4nqt"
Jul 23 15:56:07.186758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.186721    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mfwk\" (UniqueName: \"kubernetes.io/projected/9fbf1344-5b40-4019-ad13-0eacbc6469e2-kube-api-access-4mfwk\") pod \"kuadrant-operator-catalog-p4nqt\" (UID: \"9fbf1344-5b40-4019-ad13-0eacbc6469e2\") " pod="kuadrant-system/kuadrant-operator-catalog-p4nqt"
Jul 23 15:56:07.303044 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.302975    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt"
Jul 23 15:56:07.343894 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.343866    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p4nqt"]
Jul 23 15:56:07.428085 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.428057    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p4nqt"]
Jul 23 15:56:07.429506 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:56:07.429474    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbf1344_5b40_4019_ad13_0eacbc6469e2.slice/crio-9542e8e85ef3e7aea4e802bc9aecc6766daea554d037f43e6fa55d69bf2bf5ca WatchSource:0}: Error finding container 9542e8e85ef3e7aea4e802bc9aecc6766daea554d037f43e6fa55d69bf2bf5ca: Status 404 returned error can't find the container with id 9542e8e85ef3e7aea4e802bc9aecc6766daea554d037f43e6fa55d69bf2bf5ca
Jul 23 15:56:07.474133 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.474102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt" event={"ID":"9fbf1344-5b40-4019-ad13-0eacbc6469e2","Type":"ContainerStarted","Data":"9542e8e85ef3e7aea4e802bc9aecc6766daea554d037f43e6fa55d69bf2bf5ca"}
Jul 23 15:56:07.557825 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.557769    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bj7kv"]
Jul 23 15:56:07.560651 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.560636    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:07.573785 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.573765    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bj7kv"]
Jul 23 15:56:07.659086 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.659065    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjp6x\" (UniqueName: \"kubernetes.io/projected/b325de6f-af2d-438d-baa7-bcacdc58a159-kube-api-access-xjp6x\") pod \"kuadrant-operator-catalog-bj7kv\" (UID: \"b325de6f-af2d-438d-baa7-bcacdc58a159\") " pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:07.760312 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.760287    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjp6x\" (UniqueName: \"kubernetes.io/projected/b325de6f-af2d-438d-baa7-bcacdc58a159-kube-api-access-xjp6x\") pod \"kuadrant-operator-catalog-bj7kv\" (UID: \"b325de6f-af2d-438d-baa7-bcacdc58a159\") " pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:07.769820 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.769799    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjp6x\" (UniqueName: \"kubernetes.io/projected/b325de6f-af2d-438d-baa7-bcacdc58a159-kube-api-access-xjp6x\") pod \"kuadrant-operator-catalog-bj7kv\" (UID: \"b325de6f-af2d-438d-baa7-bcacdc58a159\") " pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:07.869305 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:07.869243    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:08.015721 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:08.015694    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bj7kv"]
Jul 23 15:56:08.017716 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:56:08.017686    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb325de6f_af2d_438d_baa7_bcacdc58a159.slice/crio-67150dbef04dbf1bd638d33b47ef280d076f83b3fb316150ce7b788d9bd7f40f WatchSource:0}: Error finding container 67150dbef04dbf1bd638d33b47ef280d076f83b3fb316150ce7b788d9bd7f40f: Status 404 returned error can't find the container with id 67150dbef04dbf1bd638d33b47ef280d076f83b3fb316150ce7b788d9bd7f40f
Jul 23 15:56:08.480369 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:08.480309    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv" event={"ID":"b325de6f-af2d-438d-baa7-bcacdc58a159","Type":"ContainerStarted","Data":"67150dbef04dbf1bd638d33b47ef280d076f83b3fb316150ce7b788d9bd7f40f"}
Jul 23 15:56:09.488463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.488413    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv" event={"ID":"b325de6f-af2d-438d-baa7-bcacdc58a159","Type":"ContainerStarted","Data":"14948111334abe5a1c4156c447f1c49a7fc7701123fb2afab49c39ce6fda6e66"}
Jul 23 15:56:09.490862 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.490830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt" event={"ID":"9fbf1344-5b40-4019-ad13-0eacbc6469e2","Type":"ContainerStarted","Data":"a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264"}
Jul 23 15:56:09.490997 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.490901    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt" podUID="9fbf1344-5b40-4019-ad13-0eacbc6469e2" containerName="registry-server" containerID="cri-o://a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264" gracePeriod=2
Jul 23 15:56:09.506884 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.506834    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv" podStartSLOduration=1.175579028 podStartE2EDuration="2.506820018s" totalImagesPullingTime="1.33124099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:56:09.506247863 +0000 UTC m=+457.131782341" watchObservedRunningTime="2026-07-23 15:56:09.506820018 +0000 UTC m=+457.132354493"
Jul 23 15:56:09.524501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.524456    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt" podStartSLOduration=1.605199391 podStartE2EDuration="3.524446848s" totalImagesPullingTime="1.919247457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:56:09.523436496 +0000 UTC m=+457.148970974" watchObservedRunningTime="2026-07-23 15:56:09.524446848 +0000 UTC m=+457.149981323"
Jul 23 15:56:09.723355 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.723334    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt"
Jul 23 15:56:09.777410 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.777383    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mfwk\" (UniqueName: \"kubernetes.io/projected/9fbf1344-5b40-4019-ad13-0eacbc6469e2-kube-api-access-4mfwk\") pod \"9fbf1344-5b40-4019-ad13-0eacbc6469e2\" (UID: \"9fbf1344-5b40-4019-ad13-0eacbc6469e2\") "
Jul 23 15:56:09.779407 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.779384    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbf1344-5b40-4019-ad13-0eacbc6469e2-kube-api-access-4mfwk" (OuterVolumeSpecName: "kube-api-access-4mfwk") pod "9fbf1344-5b40-4019-ad13-0eacbc6469e2" (UID: "9fbf1344-5b40-4019-ad13-0eacbc6469e2"). InnerVolumeSpecName "kube-api-access-4mfwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:56:09.878013 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:09.877961    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mfwk\" (UniqueName: \"kubernetes.io/projected/9fbf1344-5b40-4019-ad13-0eacbc6469e2-kube-api-access-4mfwk\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:56:10.496660 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.496631    2555 generic.go:356] "Generic (PLEG): container finished" podID="9fbf1344-5b40-4019-ad13-0eacbc6469e2" containerID="a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264" exitCode=0
Jul 23 15:56:10.497112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.496683    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt"
Jul 23 15:56:10.497112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.496722    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt" event={"ID":"9fbf1344-5b40-4019-ad13-0eacbc6469e2","Type":"ContainerDied","Data":"a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264"}
Jul 23 15:56:10.497112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.496794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p4nqt" event={"ID":"9fbf1344-5b40-4019-ad13-0eacbc6469e2","Type":"ContainerDied","Data":"9542e8e85ef3e7aea4e802bc9aecc6766daea554d037f43e6fa55d69bf2bf5ca"}
Jul 23 15:56:10.497112 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.496817    2555 scope.go:117] "RemoveContainer" containerID="a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264"
Jul 23 15:56:10.505607 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.505590    2555 scope.go:117] "RemoveContainer" containerID="a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264"
Jul 23 15:56:10.505874 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:56:10.505857    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264\": container with ID starting with a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264 not found: ID does not exist" containerID="a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264"
Jul 23 15:56:10.505927 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.505884    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264"} err="failed to get container status \"a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264\": rpc error: code = NotFound desc = could not find container \"a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264\": container with ID starting with a0ea88170fdef59fd84917c6cc17f9b0f40d2862f68cdb56e027d86b9644a264 not found: ID does not exist"
Jul 23 15:56:10.517849 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.517828    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p4nqt"]
Jul 23 15:56:10.520304 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.520285    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p4nqt"]
Jul 23 15:56:10.808703 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:10.808675    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fbf1344-5b40-4019-ad13-0eacbc6469e2" path="/var/lib/kubelet/pods/9fbf1344-5b40-4019-ad13-0eacbc6469e2/volumes"
Jul 23 15:56:17.869908 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:17.869877    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:17.870294 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:17.869918    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:17.891985 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:17.891964    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:18.551047 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:18.551021    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-bj7kv"
Jul 23 15:56:38.877082 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:38.877050    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"]
Jul 23 15:56:38.877465 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:38.877326    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbf1344-5b40-4019-ad13-0eacbc6469e2" containerName="registry-server"
Jul 23 15:56:38.877465 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:38.877336    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbf1344-5b40-4019-ad13-0eacbc6469e2" containerName="registry-server"
Jul 23 15:56:38.877465 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:38.877388    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbf1344-5b40-4019-ad13-0eacbc6469e2" containerName="registry-server"
Jul 23 15:56:38.881572 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:38.881555    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:56:38.884450 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:38.884432    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-48p8h\""
Jul 23 15:56:38.889626 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:38.889602    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"]
Jul 23 15:56:38.984351 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:38.984320    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45wc\" (UniqueName: \"kubernetes.io/projected/0dcd781c-4c08-4c45-8d42-9313fecf5781-kube-api-access-c45wc\") pod \"limitador-operator-controller-manager-859d7b78bd-rwktp\" (UID: \"0dcd781c-4c08-4c45-8d42-9313fecf5781\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:56:39.084916 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:39.084881    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45wc\" (UniqueName: \"kubernetes.io/projected/0dcd781c-4c08-4c45-8d42-9313fecf5781-kube-api-access-c45wc\") pod \"limitador-operator-controller-manager-859d7b78bd-rwktp\" (UID: \"0dcd781c-4c08-4c45-8d42-9313fecf5781\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:56:39.099002 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:39.098972    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45wc\" (UniqueName: \"kubernetes.io/projected/0dcd781c-4c08-4c45-8d42-9313fecf5781-kube-api-access-c45wc\") pod \"limitador-operator-controller-manager-859d7b78bd-rwktp\" (UID: \"0dcd781c-4c08-4c45-8d42-9313fecf5781\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:56:39.192320 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:39.192295    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:56:39.330929 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:39.330897    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"]
Jul 23 15:56:39.334798 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:56:39.334765    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcd781c_4c08_4c45_8d42_9313fecf5781.slice/crio-50cf3113e1ca5b64f23ce4e554f62b9754cfaf55b31f700557b1db26f74cdaa7 WatchSource:0}: Error finding container 50cf3113e1ca5b64f23ce4e554f62b9754cfaf55b31f700557b1db26f74cdaa7: Status 404 returned error can't find the container with id 50cf3113e1ca5b64f23ce4e554f62b9754cfaf55b31f700557b1db26f74cdaa7
Jul 23 15:56:39.615099 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:39.615014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp" event={"ID":"0dcd781c-4c08-4c45-8d42-9313fecf5781","Type":"ContainerStarted","Data":"50cf3113e1ca5b64f23ce4e554f62b9754cfaf55b31f700557b1db26f74cdaa7"}
Jul 23 15:56:42.628810 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:42.628775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp" event={"ID":"0dcd781c-4c08-4c45-8d42-9313fecf5781","Type":"ContainerStarted","Data":"d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648"}
Jul 23 15:56:42.629212 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:42.628879    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:56:42.698409 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:42.698358    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp" podStartSLOduration=2.34446195 podStartE2EDuration="4.698336957s" totalImagesPullingTime="2.353875007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:56:42.697240349 +0000 UTC m=+490.322774825" watchObservedRunningTime="2026-07-23 15:56:42.698336957 +0000 UTC m=+490.323871433"
Jul 23 15:56:44.555652 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.555614    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"]
Jul 23 15:56:44.563870 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.563852    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:44.585675 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.585654    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m2dnw\""
Jul 23 15:56:44.593855 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.593832    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"]
Jul 23 15:56:44.627950 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.627921    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58145f5-9bd5-459a-8afb-0813a579b0bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" (UID: \"d58145f5-9bd5-459a-8afb-0813a579b0bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:44.628116 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.627982    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7kc\" (UniqueName: \"kubernetes.io/projected/d58145f5-9bd5-459a-8afb-0813a579b0bf-kube-api-access-5p7kc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" (UID: \"d58145f5-9bd5-459a-8afb-0813a579b0bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:44.729271 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.729232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58145f5-9bd5-459a-8afb-0813a579b0bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" (UID: \"d58145f5-9bd5-459a-8afb-0813a579b0bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:44.729433 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.729289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7kc\" (UniqueName: \"kubernetes.io/projected/d58145f5-9bd5-459a-8afb-0813a579b0bf-kube-api-access-5p7kc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" (UID: \"d58145f5-9bd5-459a-8afb-0813a579b0bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:44.729603 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.729579    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58145f5-9bd5-459a-8afb-0813a579b0bf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" (UID: \"d58145f5-9bd5-459a-8afb-0813a579b0bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:44.744457 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.744428    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7kc\" (UniqueName: \"kubernetes.io/projected/d58145f5-9bd5-459a-8afb-0813a579b0bf-kube-api-access-5p7kc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" (UID: \"d58145f5-9bd5-459a-8afb-0813a579b0bf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:44.872950 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:44.872863    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:45.031483 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:45.031445    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"]
Jul 23 15:56:45.034459 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:56:45.034425    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58145f5_9bd5_459a_8afb_0813a579b0bf.slice/crio-6edda79c53ff0c4bdcd6e6be470800b3e2233541661250b6f6e05ace8e378842 WatchSource:0}: Error finding container 6edda79c53ff0c4bdcd6e6be470800b3e2233541661250b6f6e05ace8e378842: Status 404 returned error can't find the container with id 6edda79c53ff0c4bdcd6e6be470800b3e2233541661250b6f6e05ace8e378842
Jul 23 15:56:45.641326 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:45.641291    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4" event={"ID":"d58145f5-9bd5-459a-8afb-0813a579b0bf","Type":"ContainerStarted","Data":"6edda79c53ff0c4bdcd6e6be470800b3e2233541661250b6f6e05ace8e378842"}
Jul 23 15:56:48.506930 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.506885    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"]
Jul 23 15:56:48.511081 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.511048    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"
Jul 23 15:56:48.514734 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.514694    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-s9k66\""
Jul 23 15:56:48.532431 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.532405    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"]
Jul 23 15:56:48.664161 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.664118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhnf\" (UniqueName: \"kubernetes.io/projected/4b1e999c-a84e-4456-b43b-a17998c8555c-kube-api-access-bfhnf\") pod \"authorino-operator-75cc8d5cb7-pcl7c\" (UID: \"4b1e999c-a84e-4456-b43b-a17998c8555c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"
Jul 23 15:56:48.765624 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.765536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhnf\" (UniqueName: \"kubernetes.io/projected/4b1e999c-a84e-4456-b43b-a17998c8555c-kube-api-access-bfhnf\") pod \"authorino-operator-75cc8d5cb7-pcl7c\" (UID: \"4b1e999c-a84e-4456-b43b-a17998c8555c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"
Jul 23 15:56:48.801198 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.801169    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhnf\" (UniqueName: \"kubernetes.io/projected/4b1e999c-a84e-4456-b43b-a17998c8555c-kube-api-access-bfhnf\") pod \"authorino-operator-75cc8d5cb7-pcl7c\" (UID: \"4b1e999c-a84e-4456-b43b-a17998c8555c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"
Jul 23 15:56:48.823504 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.823477    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"
Jul 23 15:56:48.981404 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:48.981253    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"]
Jul 23 15:56:48.984210 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:56:48.984154    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1e999c_a84e_4456_b43b_a17998c8555c.slice/crio-19f045cf29d6cba746e70bdc5f7df01da5624c4475c3c5940b82605abfa56477 WatchSource:0}: Error finding container 19f045cf29d6cba746e70bdc5f7df01da5624c4475c3c5940b82605abfa56477: Status 404 returned error can't find the container with id 19f045cf29d6cba746e70bdc5f7df01da5624c4475c3c5940b82605abfa56477
Jul 23 15:56:49.658547 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:49.658500    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c" event={"ID":"4b1e999c-a84e-4456-b43b-a17998c8555c","Type":"ContainerStarted","Data":"19f045cf29d6cba746e70bdc5f7df01da5624c4475c3c5940b82605abfa56477"}
Jul 23 15:56:50.666130 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:50.666095    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4" event={"ID":"d58145f5-9bd5-459a-8afb-0813a579b0bf","Type":"ContainerStarted","Data":"021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec"}
Jul 23 15:56:50.666597 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:50.666211    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:56:50.709716 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:50.709626    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4" podStartSLOduration=1.275828961 podStartE2EDuration="6.70960534s" totalImagesPullingTime="5.433776379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:56:50.705148408 +0000 UTC m=+498.330682883" watchObservedRunningTime="2026-07-23 15:56:50.70960534 +0000 UTC m=+498.335139820"
Jul 23 15:56:52.676862 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:52.676823    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c" event={"ID":"4b1e999c-a84e-4456-b43b-a17998c8555c","Type":"ContainerStarted","Data":"739bb5ea64367dfcda18a966633d6516b0c021247fe05e3bbc2b857b281a1a44"}
Jul 23 15:56:52.677258 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:52.676940    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"
Jul 23 15:56:52.734510 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:52.734455    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c" podStartSLOduration=1.5488600049999999 podStartE2EDuration="4.734437889s" totalImagesPullingTime="3.185577884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:56:52.732633032 +0000 UTC m=+500.358167509" watchObservedRunningTime="2026-07-23 15:56:52.734437889 +0000 UTC m=+500.359972365"
Jul 23 15:56:53.635198 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:56:53.635162    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:01.673310 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:01.673278    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:03.684473 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.684438    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pcl7c"
Jul 23 15:57:03.828576 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.828541    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"]
Jul 23 15:57:03.828821 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.828723    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" containerName="manager" containerID="cri-o://021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec" gracePeriod=2
Jul 23 15:57:03.844564 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.844535    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"]
Jul 23 15:57:03.888341 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.888295    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"]
Jul 23 15:57:03.888551 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.888513    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" containerName="manager" containerID="cri-o://d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648" gracePeriod=2
Jul 23 15:57:03.899931 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.899905    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"]
Jul 23 15:57:03.907724 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:03.907695    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:03.910815 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.910790    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"]
Jul 23 15:57:03.911236 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.911221    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" containerName="manager"
Jul 23 15:57:03.911279 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.911240    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" containerName="manager"
Jul 23 15:57:03.911279 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.911255    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" containerName="manager"
Jul 23 15:57:03.911279 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.911264    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" containerName="manager"
Jul 23 15:57:03.911376 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.911342    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" containerName="manager"
Jul 23 15:57:03.911376 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.911355    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" containerName="manager"
Jul 23 15:57:03.914496 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.914478    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:03.936511 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:03.936430    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rwktp\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:03.942402 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.942378    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"]
Jul 23 15:57:03.976365 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.976335    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"]
Jul 23 15:57:03.979640 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:03.979617    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"
Jul 23 15:57:03.995968 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:03.995931    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:04.006611 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.006562    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"]
Jul 23 15:57:04.065697 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.065641    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rwktp\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:04.067675 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.067650    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:04.095161 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.095121    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzhf\" (UniqueName: \"kubernetes.io/projected/4fc6a354-0b29-4768-b027-eba50e0fc006-kube-api-access-xlzhf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bqklx\" (UID: \"4fc6a354-0b29-4768-b027-eba50e0fc006\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:04.095286 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.095185    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fc6a354-0b29-4768-b027-eba50e0fc006-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bqklx\" (UID: \"4fc6a354-0b29-4768-b027-eba50e0fc006\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:04.095329 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.095302    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fd6z\" (UniqueName: \"kubernetes.io/projected/d44de4d5-7b88-4968-bef2-92f432ed3a11-kube-api-access-2fd6z\") pod \"limitador-operator-controller-manager-859d7b78bd-7nndx\" (UID: \"d44de4d5-7b88-4968-bef2-92f432ed3a11\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"
Jul 23 15:57:04.144247 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.144223    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:04.147666 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.147633    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rwktp\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:04.148118 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.148101    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:04.150709 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.150674    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:04.166400 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.166371    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rwktp\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:04.186110 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.186083    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:04.196497 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.196445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fd6z\" (UniqueName: \"kubernetes.io/projected/d44de4d5-7b88-4968-bef2-92f432ed3a11-kube-api-access-2fd6z\") pod \"limitador-operator-controller-manager-859d7b78bd-7nndx\" (UID: \"d44de4d5-7b88-4968-bef2-92f432ed3a11\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"
Jul 23 15:57:04.196497 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.196478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzhf\" (UniqueName: \"kubernetes.io/projected/4fc6a354-0b29-4768-b027-eba50e0fc006-kube-api-access-xlzhf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bqklx\" (UID: \"4fc6a354-0b29-4768-b027-eba50e0fc006\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:04.196616 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.196506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fc6a354-0b29-4768-b027-eba50e0fc006-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bqklx\" (UID: \"4fc6a354-0b29-4768-b027-eba50e0fc006\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:04.196915 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.196899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fc6a354-0b29-4768-b027-eba50e0fc006-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bqklx\" (UID: \"4fc6a354-0b29-4768-b027-eba50e0fc006\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:04.224404 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.224376    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzhf\" (UniqueName: \"kubernetes.io/projected/4fc6a354-0b29-4768-b027-eba50e0fc006-kube-api-access-xlzhf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bqklx\" (UID: \"4fc6a354-0b29-4768-b027-eba50e0fc006\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:04.224404 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.224385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fd6z\" (UniqueName: \"kubernetes.io/projected/d44de4d5-7b88-4968-bef2-92f432ed3a11-kube-api-access-2fd6z\") pod \"limitador-operator-controller-manager-859d7b78bd-7nndx\" (UID: \"d44de4d5-7b88-4968-bef2-92f432ed3a11\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"
Jul 23 15:57:04.256226 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.256199    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:04.289245 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.289216    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"
Jul 23 15:57:04.297515 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.297486    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c45wc\" (UniqueName: \"kubernetes.io/projected/0dcd781c-4c08-4c45-8d42-9313fecf5781-kube-api-access-c45wc\") pod \"0dcd781c-4c08-4c45-8d42-9313fecf5781\" (UID: \"0dcd781c-4c08-4c45-8d42-9313fecf5781\") "
Jul 23 15:57:04.297625 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.297529    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p7kc\" (UniqueName: \"kubernetes.io/projected/d58145f5-9bd5-459a-8afb-0813a579b0bf-kube-api-access-5p7kc\") pod \"d58145f5-9bd5-459a-8afb-0813a579b0bf\" (UID: \"d58145f5-9bd5-459a-8afb-0813a579b0bf\") "
Jul 23 15:57:04.297625 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.297594    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58145f5-9bd5-459a-8afb-0813a579b0bf-extensions-socket-volume\") pod \"d58145f5-9bd5-459a-8afb-0813a579b0bf\" (UID: \"d58145f5-9bd5-459a-8afb-0813a579b0bf\") "
Jul 23 15:57:04.298117 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.298088    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58145f5-9bd5-459a-8afb-0813a579b0bf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d58145f5-9bd5-459a-8afb-0813a579b0bf" (UID: "d58145f5-9bd5-459a-8afb-0813a579b0bf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:57:04.299609 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.299583    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcd781c-4c08-4c45-8d42-9313fecf5781-kube-api-access-c45wc" (OuterVolumeSpecName: "kube-api-access-c45wc") pod "0dcd781c-4c08-4c45-8d42-9313fecf5781" (UID: "0dcd781c-4c08-4c45-8d42-9313fecf5781"). InnerVolumeSpecName "kube-api-access-c45wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:57:04.299609 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.299588    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58145f5-9bd5-459a-8afb-0813a579b0bf-kube-api-access-5p7kc" (OuterVolumeSpecName: "kube-api-access-5p7kc") pod "d58145f5-9bd5-459a-8afb-0813a579b0bf" (UID: "d58145f5-9bd5-459a-8afb-0813a579b0bf"). InnerVolumeSpecName "kube-api-access-5p7kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:57:04.398501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.398468    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d58145f5-9bd5-459a-8afb-0813a579b0bf-extensions-socket-volume\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:57:04.398501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.398498    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c45wc\" (UniqueName: \"kubernetes.io/projected/0dcd781c-4c08-4c45-8d42-9313fecf5781-kube-api-access-c45wc\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:57:04.398501 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.398510    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7kc\" (UniqueName: \"kubernetes.io/projected/d58145f5-9bd5-459a-8afb-0813a579b0bf-kube-api-access-5p7kc\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:57:04.412851 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.412797    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"]
Jul 23 15:57:04.415072 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:57:04.415032    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc6a354_0b29_4768_b027_eba50e0fc006.slice/crio-8e84d9e51e7fde88aeedffd5e09108e4974aa722ed26d96445f31fd4c9466825 WatchSource:0}: Error finding container 8e84d9e51e7fde88aeedffd5e09108e4974aa722ed26d96445f31fd4c9466825: Status 404 returned error can't find the container with id 8e84d9e51e7fde88aeedffd5e09108e4974aa722ed26d96445f31fd4c9466825
Jul 23 15:57:04.461537 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.461510    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"]
Jul 23 15:57:04.464553 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:57:04.464517    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44de4d5_7b88_4968_bef2_92f432ed3a11.slice/crio-8b653cf77d23b36802718cd501a90a40e6955f823766a831343e0f79e21b1a2e WatchSource:0}: Error finding container 8b653cf77d23b36802718cd501a90a40e6955f823766a831343e0f79e21b1a2e: Status 404 returned error can't find the container with id 8b653cf77d23b36802718cd501a90a40e6955f823766a831343e0f79e21b1a2e
Jul 23 15:57:04.733229 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.733147    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx" event={"ID":"4fc6a354-0b29-4768-b027-eba50e0fc006","Type":"ContainerStarted","Data":"d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043"}
Jul 23 15:57:04.733229 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.733204    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx" event={"ID":"4fc6a354-0b29-4768-b027-eba50e0fc006","Type":"ContainerStarted","Data":"8e84d9e51e7fde88aeedffd5e09108e4974aa722ed26d96445f31fd4c9466825"}
Jul 23 15:57:04.733657 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.733333    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:04.734450 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.734432    2555 generic.go:356] "Generic (PLEG): container finished" podID="0dcd781c-4c08-4c45-8d42-9313fecf5781" containerID="d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648" exitCode=0
Jul 23 15:57:04.734511 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.734477    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:04.734546 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.734513    2555 scope.go:117] "RemoveContainer" containerID="d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648"
Jul 23 15:57:04.735921 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.735898    2555 generic.go:356] "Generic (PLEG): container finished" podID="d58145f5-9bd5-459a-8afb-0813a579b0bf" containerID="021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec" exitCode=0
Jul 23 15:57:04.736047 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.735938    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:04.737850 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.737825    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx" event={"ID":"d44de4d5-7b88-4968-bef2-92f432ed3a11","Type":"ContainerStarted","Data":"ebb23c98be58e171291956a768b226ee49d60e9df8da749a399c8d028712803f"}
Jul 23 15:57:04.737958 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.737871    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx" event={"ID":"d44de4d5-7b88-4968-bef2-92f432ed3a11","Type":"ContainerStarted","Data":"8b653cf77d23b36802718cd501a90a40e6955f823766a831343e0f79e21b1a2e"}
Jul 23 15:57:04.737958 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.737899    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"
Jul 23 15:57:04.740048 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.740024    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rwktp\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:04.743909 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.743893    2555 scope.go:117] "RemoveContainer" containerID="d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648"
Jul 23 15:57:04.744200 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.744180    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648\": container with ID starting with d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648 not found: ID does not exist" containerID="d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648"
Jul 23 15:57:04.744290 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.744213    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648"} err="failed to get container status \"d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648\": rpc error: code = NotFound desc = could not find container \"d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648\": container with ID starting with d7b684f276adbf8013da1c55617463d730bba7ac76d92742e812831bd3a29648 not found: ID does not exist"
Jul 23 15:57:04.744290 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.744259    2555 scope.go:117] "RemoveContainer" containerID="021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec"
Jul 23 15:57:04.752030 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.751987    2555 scope.go:117] "RemoveContainer" containerID="021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec"
Jul 23 15:57:04.752250 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.752233    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec\": container with ID starting with 021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec not found: ID does not exist" containerID="021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec"
Jul 23 15:57:04.752313 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.752259    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec"} err="failed to get container status \"021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec\": rpc error: code = NotFound desc = could not find container \"021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec\": container with ID starting with 021141fac050a368ef20317aee12bfc27e50c669dfe7fa465283eb4fc485fbec not found: ID does not exist"
Jul 23 15:57:04.792581 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.792542    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx" podStartSLOduration=1.792530111 podStartE2EDuration="1.792530111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:57:04.792246096 +0000 UTC m=+512.417780585" watchObservedRunningTime="2026-07-23 15:57:04.792530111 +0000 UTC m=+512.418064586"
Jul 23 15:57:04.798216 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.798193    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:04.805196 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.805170    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4cbx4\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4cbx4"
Jul 23 15:57:04.809450 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.809429    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" path="/var/lib/kubelet/pods/0dcd781c-4c08-4c45-8d42-9313fecf5781/volumes"
Jul 23 15:57:04.809758 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.809734    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d58145f5-9bd5-459a-8afb-0813a579b0bf" path="/var/lib/kubelet/pods/d58145f5-9bd5-459a-8afb-0813a579b0bf/volumes"
Jul 23 15:57:04.852241 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:04.852208    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx" podStartSLOduration=1.8521956450000001 podStartE2EDuration="1.852195645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:57:04.850985183 +0000 UTC m=+512.476519659" watchObservedRunningTime="2026-07-23 15:57:04.852195645 +0000 UTC m=+512.477730121"
Jul 23 15:57:04.856210 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:04.856186    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rwktp\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="0dcd781c-4c08-4c45-8d42-9313fecf5781" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rwktp"
Jul 23 15:57:15.746268 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:15.746238    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7nndx"
Jul 23 15:57:15.746605 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:15.746309    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:32.897572 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:32.897532    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"]
Jul 23 15:57:32.898013 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:32.897866    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx" podUID="4fc6a354-0b29-4768-b027-eba50e0fc006" containerName="manager" containerID="cri-o://d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043" gracePeriod=10
Jul 23 15:57:33.153658 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.153635    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:33.205283 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.205258    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlzhf\" (UniqueName: \"kubernetes.io/projected/4fc6a354-0b29-4768-b027-eba50e0fc006-kube-api-access-xlzhf\") pod \"4fc6a354-0b29-4768-b027-eba50e0fc006\" (UID: \"4fc6a354-0b29-4768-b027-eba50e0fc006\") "
Jul 23 15:57:33.205422 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.205323    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fc6a354-0b29-4768-b027-eba50e0fc006-extensions-socket-volume\") pod \"4fc6a354-0b29-4768-b027-eba50e0fc006\" (UID: \"4fc6a354-0b29-4768-b027-eba50e0fc006\") "
Jul 23 15:57:33.205654 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.205632    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc6a354-0b29-4768-b027-eba50e0fc006-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4fc6a354-0b29-4768-b027-eba50e0fc006" (UID: "4fc6a354-0b29-4768-b027-eba50e0fc006"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 15:57:33.207383 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.207360    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc6a354-0b29-4768-b027-eba50e0fc006-kube-api-access-xlzhf" (OuterVolumeSpecName: "kube-api-access-xlzhf") pod "4fc6a354-0b29-4768-b027-eba50e0fc006" (UID: "4fc6a354-0b29-4768-b027-eba50e0fc006"). InnerVolumeSpecName "kube-api-access-xlzhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:57:33.306225 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.306169    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzhf\" (UniqueName: \"kubernetes.io/projected/4fc6a354-0b29-4768-b027-eba50e0fc006-kube-api-access-xlzhf\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:57:33.306225 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.306189    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fc6a354-0b29-4768-b027-eba50e0fc006-extensions-socket-volume\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:57:33.866871 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.866836    2555 generic.go:356] "Generic (PLEG): container finished" podID="4fc6a354-0b29-4768-b027-eba50e0fc006" containerID="d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043" exitCode=0
Jul 23 15:57:33.867081 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.866883    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"
Jul 23 15:57:33.867081 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.866896    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx" event={"ID":"4fc6a354-0b29-4768-b027-eba50e0fc006","Type":"ContainerDied","Data":"d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043"}
Jul 23 15:57:33.867081 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.866947    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx" event={"ID":"4fc6a354-0b29-4768-b027-eba50e0fc006","Type":"ContainerDied","Data":"8e84d9e51e7fde88aeedffd5e09108e4974aa722ed26d96445f31fd4c9466825"}
Jul 23 15:57:33.867081 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.866976    2555 scope.go:117] "RemoveContainer" containerID="d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043"
Jul 23 15:57:33.877556 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.877384    2555 scope.go:117] "RemoveContainer" containerID="d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043"
Jul 23 15:57:33.877850 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:57:33.877821    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043\": container with ID starting with d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043 not found: ID does not exist" containerID="d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043"
Jul 23 15:57:33.877940 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.877856    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043"} err="failed to get container status \"d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043\": rpc error: code = NotFound desc = could not find container \"d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043\": container with ID starting with d8c14481bf4a5535e6078264cb00727f1a304538a4657cd88be0f87ff8c02043 not found: ID does not exist"
Jul 23 15:57:33.913635 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.913606    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"]
Jul 23 15:57:33.920131 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:33.920112    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bqklx"]
Jul 23 15:57:34.809014 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:34.808985    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc6a354-0b29-4768-b027-eba50e0fc006" path="/var/lib/kubelet/pods/4fc6a354-0b29-4768-b027-eba50e0fc006/volumes"
Jul 23 15:57:49.335032 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.335005    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"]
Jul 23 15:57:49.335404 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.335302    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc6a354-0b29-4768-b027-eba50e0fc006" containerName="manager"
Jul 23 15:57:49.335404 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.335313    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc6a354-0b29-4768-b027-eba50e0fc006" containerName="manager"
Jul 23 15:57:49.335404 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.335362    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc6a354-0b29-4768-b027-eba50e0fc006" containerName="manager"
Jul 23 15:57:49.339815 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.339797    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.343172 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.343154    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wxddq\""
Jul 23 15:57:49.350639 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.350614    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"]
Jul 23 15:57:49.426163 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.426296 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.426296 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.426296 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c022797-c084-4d04-ac20-05fdf9f9d87b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.426296 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.426296 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.426296 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426295    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.426499 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kpp\" (UniqueName: \"kubernetes.io/projected/1c022797-c084-4d04-ac20-05fdf9f9d87b-kube-api-access-74kpp\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.426499 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.426353    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.527662 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.527631    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.527662 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.527663    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.527844 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.527687    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.527971 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.527952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c022797-c084-4d04-ac20-05fdf9f9d87b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528019 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.527990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528059 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528019    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528107 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528157 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528016    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528157 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528115    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kpp\" (UniqueName: \"kubernetes.io/projected/1c022797-c084-4d04-ac20-05fdf9f9d87b-kube-api-access-74kpp\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528157 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528130    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528157 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528146    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528472 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528652 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528635    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.528731 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.528717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c022797-c084-4d04-ac20-05fdf9f9d87b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.530626 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.530598    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.530857 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.530836    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.540471 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.540448    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c022797-c084-4d04-ac20-05fdf9f9d87b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.540554 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.540503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kpp\" (UniqueName: \"kubernetes.io/projected/1c022797-c084-4d04-ac20-05fdf9f9d87b-kube-api-access-74kpp\") pod \"maas-default-gateway-openshift-default-56654dcf7-rp4xm\" (UID: \"1c022797-c084-4d04-ac20-05fdf9f9d87b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.651007 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.650961    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:49.783024 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.782996    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"]
Jul 23 15:57:49.787131 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:57:49.787108    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c022797_c084_4d04_ac20_05fdf9f9d87b.slice/crio-9dceff7b247a7a76b2bdbf33dd82eb2f070a647c559262c08158ebcc13b25603 WatchSource:0}: Error finding container 9dceff7b247a7a76b2bdbf33dd82eb2f070a647c559262c08158ebcc13b25603: Status 404 returned error can't find the container with id 9dceff7b247a7a76b2bdbf33dd82eb2f070a647c559262c08158ebcc13b25603
Jul 23 15:57:49.789127 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.789098    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:57:49.789197 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.789159    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:57:49.789197 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.789191    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:57:49.934740 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.934654    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" event={"ID":"1c022797-c084-4d04-ac20-05fdf9f9d87b","Type":"ContainerStarted","Data":"71f4cd9843cf1daeaa54f902a9debfb9f0c13cb322ee59f957c5f4847851d776"}
Jul 23 15:57:49.934740 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.934722    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" event={"ID":"1c022797-c084-4d04-ac20-05fdf9f9d87b","Type":"ContainerStarted","Data":"9dceff7b247a7a76b2bdbf33dd82eb2f070a647c559262c08158ebcc13b25603"}
Jul 23 15:57:49.958300 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:49.958247    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" podStartSLOduration=0.958231541 podStartE2EDuration="958.231541ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:57:49.956224336 +0000 UTC m=+557.581758821" watchObservedRunningTime="2026-07-23 15:57:49.958231541 +0000 UTC m=+557.583766018"
Jul 23 15:57:50.651260 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:50.651229    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:50.655903 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:50.655880    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:50.939339 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:50.939262    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:50.940002 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:50.939986    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:57:52.746249 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.746217    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 15:57:52.749621 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.749602    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:52.752111 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.752080    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 15:57:52.752212 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.752140    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8257l\""
Jul 23 15:57:52.759954 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.759932    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 15:57:52.798089 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.798055    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 15:57:52.856368 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.856339    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a3b6746f-21c1-42d7-a41f-b8e107a06389-config-file\") pod \"limitador-limitador-54d6d988d5-7tk7h\" (UID: \"a3b6746f-21c1-42d7-a41f-b8e107a06389\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:52.856484 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.856379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cz4g\" (UniqueName: \"kubernetes.io/projected/a3b6746f-21c1-42d7-a41f-b8e107a06389-kube-api-access-5cz4g\") pod \"limitador-limitador-54d6d988d5-7tk7h\" (UID: \"a3b6746f-21c1-42d7-a41f-b8e107a06389\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:52.957247 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.957221    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cz4g\" (UniqueName: \"kubernetes.io/projected/a3b6746f-21c1-42d7-a41f-b8e107a06389-kube-api-access-5cz4g\") pod \"limitador-limitador-54d6d988d5-7tk7h\" (UID: \"a3b6746f-21c1-42d7-a41f-b8e107a06389\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:52.957366 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.957296    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a3b6746f-21c1-42d7-a41f-b8e107a06389-config-file\") pod \"limitador-limitador-54d6d988d5-7tk7h\" (UID: \"a3b6746f-21c1-42d7-a41f-b8e107a06389\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:52.957844 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.957827    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a3b6746f-21c1-42d7-a41f-b8e107a06389-config-file\") pod \"limitador-limitador-54d6d988d5-7tk7h\" (UID: \"a3b6746f-21c1-42d7-a41f-b8e107a06389\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:52.968463 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:52.968433    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cz4g\" (UniqueName: \"kubernetes.io/projected/a3b6746f-21c1-42d7-a41f-b8e107a06389-kube-api-access-5cz4g\") pod \"limitador-limitador-54d6d988d5-7tk7h\" (UID: \"a3b6746f-21c1-42d7-a41f-b8e107a06389\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:53.060109 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.060061    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:53.193002 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.192977    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 15:57:53.195302 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:57:53.195275    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b6746f_21c1_42d7_a41f_b8e107a06389.slice/crio-c2841b1e9b3b7388c6a8fcaf8a01ae3ad2bbdf8aa9ad6d9a6b1bac937fec84fe WatchSource:0}: Error finding container c2841b1e9b3b7388c6a8fcaf8a01ae3ad2bbdf8aa9ad6d9a6b1bac937fec84fe: Status 404 returned error can't find the container with id c2841b1e9b3b7388c6a8fcaf8a01ae3ad2bbdf8aa9ad6d9a6b1bac937fec84fe
Jul 23 15:57:53.254717 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.254690    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-ncbg4"]
Jul 23 15:57:53.258956 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.258938    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ncbg4"
Jul 23 15:57:53.261492 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.261473    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dxcfw\""
Jul 23 15:57:53.267709 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.267689    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ncbg4"]
Jul 23 15:57:53.359674 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.359622    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvsl\" (UniqueName: \"kubernetes.io/projected/108f9253-af3b-4617-b29d-8bf050fcd694-kube-api-access-scvsl\") pod \"authorino-58cd955675-ncbg4\" (UID: \"108f9253-af3b-4617-b29d-8bf050fcd694\") " pod="kuadrant-system/authorino-58cd955675-ncbg4"
Jul 23 15:57:53.460506 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.460482    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvsl\" (UniqueName: \"kubernetes.io/projected/108f9253-af3b-4617-b29d-8bf050fcd694-kube-api-access-scvsl\") pod \"authorino-58cd955675-ncbg4\" (UID: \"108f9253-af3b-4617-b29d-8bf050fcd694\") " pod="kuadrant-system/authorino-58cd955675-ncbg4"
Jul 23 15:57:53.469464 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.469437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvsl\" (UniqueName: \"kubernetes.io/projected/108f9253-af3b-4617-b29d-8bf050fcd694-kube-api-access-scvsl\") pod \"authorino-58cd955675-ncbg4\" (UID: \"108f9253-af3b-4617-b29d-8bf050fcd694\") " pod="kuadrant-system/authorino-58cd955675-ncbg4"
Jul 23 15:57:53.547195 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.547168    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-cpk6b"]
Jul 23 15:57:53.550395 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.550378    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cpk6b"
Jul 23 15:57:53.558970 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.558947    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cpk6b"]
Jul 23 15:57:53.568102 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.568075    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ncbg4"
Jul 23 15:57:53.662840 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.662814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnmf8\" (UniqueName: \"kubernetes.io/projected/9df83039-7476-4a24-81b9-7c122abf03f9-kube-api-access-tnmf8\") pod \"authorino-75d556bd77-cpk6b\" (UID: \"9df83039-7476-4a24-81b9-7c122abf03f9\") " pod="kuadrant-system/authorino-75d556bd77-cpk6b"
Jul 23 15:57:53.696238 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.696204    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ncbg4"]
Jul 23 15:57:53.699337 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:57:53.699313    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod108f9253_af3b_4617_b29d_8bf050fcd694.slice/crio-e798573c83f7e3caa047e2c4c8e68e0d0b01d03ad9487d67a11e6468cc9e809b WatchSource:0}: Error finding container e798573c83f7e3caa047e2c4c8e68e0d0b01d03ad9487d67a11e6468cc9e809b: Status 404 returned error can't find the container with id e798573c83f7e3caa047e2c4c8e68e0d0b01d03ad9487d67a11e6468cc9e809b
Jul 23 15:57:53.764273 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.764244    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnmf8\" (UniqueName: \"kubernetes.io/projected/9df83039-7476-4a24-81b9-7c122abf03f9-kube-api-access-tnmf8\") pod \"authorino-75d556bd77-cpk6b\" (UID: \"9df83039-7476-4a24-81b9-7c122abf03f9\") " pod="kuadrant-system/authorino-75d556bd77-cpk6b"
Jul 23 15:57:53.773519 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.773497    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnmf8\" (UniqueName: \"kubernetes.io/projected/9df83039-7476-4a24-81b9-7c122abf03f9-kube-api-access-tnmf8\") pod \"authorino-75d556bd77-cpk6b\" (UID: \"9df83039-7476-4a24-81b9-7c122abf03f9\") " pod="kuadrant-system/authorino-75d556bd77-cpk6b"
Jul 23 15:57:53.860592 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.860562    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cpk6b"
Jul 23 15:57:53.953696 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.953654    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ncbg4" event={"ID":"108f9253-af3b-4617-b29d-8bf050fcd694","Type":"ContainerStarted","Data":"e798573c83f7e3caa047e2c4c8e68e0d0b01d03ad9487d67a11e6468cc9e809b"}
Jul 23 15:57:53.955170 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.955144    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h" event={"ID":"a3b6746f-21c1-42d7-a41f-b8e107a06389","Type":"ContainerStarted","Data":"c2841b1e9b3b7388c6a8fcaf8a01ae3ad2bbdf8aa9ad6d9a6b1bac937fec84fe"}
Jul 23 15:57:53.992540 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:53.992503    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cpk6b"]
Jul 23 15:57:53.994475 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:57:53.994453    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9df83039_7476_4a24_81b9_7c122abf03f9.slice/crio-8ab95482dac5c6993dd6182517f317dd46f50aa8322c92cfc97e059f8b1ca22a WatchSource:0}: Error finding container 8ab95482dac5c6993dd6182517f317dd46f50aa8322c92cfc97e059f8b1ca22a: Status 404 returned error can't find the container with id 8ab95482dac5c6993dd6182517f317dd46f50aa8322c92cfc97e059f8b1ca22a
Jul 23 15:57:54.961989 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:54.961954    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cpk6b" event={"ID":"9df83039-7476-4a24-81b9-7c122abf03f9","Type":"ContainerStarted","Data":"8ab95482dac5c6993dd6182517f317dd46f50aa8322c92cfc97e059f8b1ca22a"}
Jul 23 15:57:56.976949 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:56.976871    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h" event={"ID":"a3b6746f-21c1-42d7-a41f-b8e107a06389","Type":"ContainerStarted","Data":"926dd4756509e727c3c5f3bba6982e3c198fca28834f9feec6a748846a92fe57"}
Jul 23 15:57:56.977375 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:56.977021    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:57:56.997379 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:56.997320    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h" podStartSLOduration=1.484716839 podStartE2EDuration="4.997296436s" totalImagesPullingTime="3.512579597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:57:56.994888322 +0000 UTC m=+564.620422813" watchObservedRunningTime="2026-07-23 15:57:56.997296436 +0000 UTC m=+564.622830913"
Jul 23 15:57:58.987428 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:58.987394    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ncbg4" event={"ID":"108f9253-af3b-4617-b29d-8bf050fcd694","Type":"ContainerStarted","Data":"f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee"}
Jul 23 15:57:58.988909 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:58.988885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cpk6b" event={"ID":"9df83039-7476-4a24-81b9-7c122abf03f9","Type":"ContainerStarted","Data":"d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b"}
Jul 23 15:57:59.003373 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:59.003325    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-ncbg4" podStartSLOduration=1.08948063 podStartE2EDuration="6.003310698s" totalImagesPullingTime="4.913830068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:57:59.001282293 +0000 UTC m=+566.626816799" watchObservedRunningTime="2026-07-23 15:57:59.003310698 +0000 UTC m=+566.628845204"
Jul 23 15:57:59.017712 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:59.017657    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-cpk6b" podStartSLOduration=1.384056734 podStartE2EDuration="6.017642253s" totalImagesPullingTime="4.633585519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:57:59.016565567 +0000 UTC m=+566.642100039" watchObservedRunningTime="2026-07-23 15:57:59.017642253 +0000 UTC m=+566.643176732"
Jul 23 15:57:59.046909 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:57:59.046880    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ncbg4"]
Jul 23 15:58:00.996915 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:00.996876    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-ncbg4" podUID="108f9253-af3b-4617-b29d-8bf050fcd694" containerName="authorino" containerID="cri-o://f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee" gracePeriod=30
Jul 23 15:58:01.256508 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:01.256450    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ncbg4"
Jul 23 15:58:01.334066 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:01.334043    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scvsl\" (UniqueName: \"kubernetes.io/projected/108f9253-af3b-4617-b29d-8bf050fcd694-kube-api-access-scvsl\") pod \"108f9253-af3b-4617-b29d-8bf050fcd694\" (UID: \"108f9253-af3b-4617-b29d-8bf050fcd694\") "
Jul 23 15:58:01.335971 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:01.335940    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108f9253-af3b-4617-b29d-8bf050fcd694-kube-api-access-scvsl" (OuterVolumeSpecName: "kube-api-access-scvsl") pod "108f9253-af3b-4617-b29d-8bf050fcd694" (UID: "108f9253-af3b-4617-b29d-8bf050fcd694"). InnerVolumeSpecName "kube-api-access-scvsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:58:01.435091 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:01.435065    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvsl\" (UniqueName: \"kubernetes.io/projected/108f9253-af3b-4617-b29d-8bf050fcd694-kube-api-access-scvsl\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:58:02.001917 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.001882    2555 generic.go:356] "Generic (PLEG): container finished" podID="108f9253-af3b-4617-b29d-8bf050fcd694" containerID="f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee" exitCode=0
Jul 23 15:58:02.002288 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.001930    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ncbg4"
Jul 23 15:58:02.002288 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.001940    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ncbg4" event={"ID":"108f9253-af3b-4617-b29d-8bf050fcd694","Type":"ContainerDied","Data":"f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee"}
Jul 23 15:58:02.002288 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.001981    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ncbg4" event={"ID":"108f9253-af3b-4617-b29d-8bf050fcd694","Type":"ContainerDied","Data":"e798573c83f7e3caa047e2c4c8e68e0d0b01d03ad9487d67a11e6468cc9e809b"}
Jul 23 15:58:02.002288 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.002001    2555 scope.go:117] "RemoveContainer" containerID="f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee"
Jul 23 15:58:02.010791 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.010742    2555 scope.go:117] "RemoveContainer" containerID="f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee"
Jul 23 15:58:02.011039 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:58:02.011020    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee\": container with ID starting with f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee not found: ID does not exist" containerID="f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee"
Jul 23 15:58:02.011082 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.011047    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee"} err="failed to get container status \"f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee\": rpc error: code = NotFound desc = could not find container \"f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee\": container with ID starting with f3de9a7fc2e1cfc21ec15c3dd4ea4112547b532457bef422a708344819d7f5ee not found: ID does not exist"
Jul 23 15:58:02.024156 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.024133    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ncbg4"]
Jul 23 15:58:02.031203 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.031182    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-ncbg4"]
Jul 23 15:58:02.810776 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:02.810731    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="108f9253-af3b-4617-b29d-8bf050fcd694" path="/var/lib/kubelet/pods/108f9253-af3b-4617-b29d-8bf050fcd694/volumes"
Jul 23 15:58:07.983046 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:07.983020    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"
Jul 23 15:58:26.053127 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.053082    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sk5fq"]
Jul 23 15:58:26.053588 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.053571    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108f9253-af3b-4617-b29d-8bf050fcd694" containerName="authorino"
Jul 23 15:58:26.053652 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.053591    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="108f9253-af3b-4617-b29d-8bf050fcd694" containerName="authorino"
Jul 23 15:58:26.053703 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.053668    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="108f9253-af3b-4617-b29d-8bf050fcd694" containerName="authorino"
Jul 23 15:58:26.057536 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.057515    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sk5fq"
Jul 23 15:58:26.069368 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.069344    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sk5fq"]
Jul 23 15:58:26.222179 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.222150    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmcpt\" (UniqueName: \"kubernetes.io/projected/b389f46f-c69a-44a9-bc5a-6a70684a6dc9-kube-api-access-pmcpt\") pod \"authorino-5d4b67c446-sk5fq\" (UID: \"b389f46f-c69a-44a9-bc5a-6a70684a6dc9\") " pod="kuadrant-system/authorino-5d4b67c446-sk5fq"
Jul 23 15:58:26.256609 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.256579    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sk5fq"]
Jul 23 15:58:26.256869 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:58:26.256838    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pmcpt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-sk5fq" podUID="b389f46f-c69a-44a9-bc5a-6a70684a6dc9"
Jul 23 15:58:26.286803 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.286774    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-xb7b6"]
Jul 23 15:58:26.290014 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.289997    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:26.292527 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.292511    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 23 15:58:26.301192 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.301172    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-xb7b6"]
Jul 23 15:58:26.323184 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.323131    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcpt\" (UniqueName: \"kubernetes.io/projected/b389f46f-c69a-44a9-bc5a-6a70684a6dc9-kube-api-access-pmcpt\") pod \"authorino-5d4b67c446-sk5fq\" (UID: \"b389f46f-c69a-44a9-bc5a-6a70684a6dc9\") " pod="kuadrant-system/authorino-5d4b67c446-sk5fq"
Jul 23 15:58:26.330711 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.330686    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-xb7b6"]
Jul 23 15:58:26.330930 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:58:26.330911    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-glv25 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-xb7b6" podUID="e7165f08-e9a2-45bb-b4ac-996f6921c37d"
Jul 23 15:58:26.337822 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.337801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcpt\" (UniqueName: \"kubernetes.io/projected/b389f46f-c69a-44a9-bc5a-6a70684a6dc9-kube-api-access-pmcpt\") pod \"authorino-5d4b67c446-sk5fq\" (UID: \"b389f46f-c69a-44a9-bc5a-6a70684a6dc9\") " pod="kuadrant-system/authorino-5d4b67c446-sk5fq"
Jul 23 15:58:26.364942 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.364918    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64bd574ffb-6qj68"]
Jul 23 15:58:26.368141 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.368125    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 15:58:26.381294 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.378945    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64bd574ffb-6qj68"]
Jul 23 15:58:26.424093 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.424068    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7165f08-e9a2-45bb-b4ac-996f6921c37d-tls-cert\") pod \"authorino-76877994-xb7b6\" (UID: \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\") " pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:26.424198 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.424097    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glv25\" (UniqueName: \"kubernetes.io/projected/e7165f08-e9a2-45bb-b4ac-996f6921c37d-kube-api-access-glv25\") pod \"authorino-76877994-xb7b6\" (UID: \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\") " pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:26.524841 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.524813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7165f08-e9a2-45bb-b4ac-996f6921c37d-tls-cert\") pod \"authorino-76877994-xb7b6\" (UID: \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\") " pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:26.524943 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.524843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glv25\" (UniqueName: \"kubernetes.io/projected/e7165f08-e9a2-45bb-b4ac-996f6921c37d-kube-api-access-glv25\") pod \"authorino-76877994-xb7b6\" (UID: \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\") " pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:26.524943 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.524880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54gn\" (UniqueName: \"kubernetes.io/projected/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-kube-api-access-g54gn\") pod \"authorino-64bd574ffb-6qj68\" (UID: \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\") " pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 15:58:26.524943 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.524922    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-tls-cert\") pod \"authorino-64bd574ffb-6qj68\" (UID: \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\") " pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 15:58:26.527312 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.527287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7165f08-e9a2-45bb-b4ac-996f6921c37d-tls-cert\") pod \"authorino-76877994-xb7b6\" (UID: \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\") " pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:26.533831 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.533809    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glv25\" (UniqueName: \"kubernetes.io/projected/e7165f08-e9a2-45bb-b4ac-996f6921c37d-kube-api-access-glv25\") pod \"authorino-76877994-xb7b6\" (UID: \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\") " pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:26.625287 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.625234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54gn\" (UniqueName: \"kubernetes.io/projected/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-kube-api-access-g54gn\") pod \"authorino-64bd574ffb-6qj68\" (UID: \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\") " pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 15:58:26.625287 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.625271    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-tls-cert\") pod \"authorino-64bd574ffb-6qj68\" (UID: \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\") " pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 15:58:26.627365 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.627347    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-tls-cert\") pod \"authorino-64bd574ffb-6qj68\" (UID: \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\") " pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 15:58:26.633308 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.633284    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54gn\" (UniqueName: \"kubernetes.io/projected/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-kube-api-access-g54gn\") pod \"authorino-64bd574ffb-6qj68\" (UID: \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\") " pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 15:58:26.676808 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.676780    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 15:58:26.808505 ip-10-0-128-121 kubenswrapper[2555]: W0723 15:58:26.808478    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod807971a2_a2e8_43d6_bb8f_26a4f2b6b2bf.slice/crio-d05063308262ea235e7286d287e014a23517a43bc57a9b2a5855781befd8181c WatchSource:0}: Error finding container d05063308262ea235e7286d287e014a23517a43bc57a9b2a5855781befd8181c: Status 404 returned error can't find the container with id d05063308262ea235e7286d287e014a23517a43bc57a9b2a5855781befd8181c
Jul 23 15:58:26.809628 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:26.809606    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64bd574ffb-6qj68"]
Jul 23 15:58:27.108298 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.108263    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64bd574ffb-6qj68" event={"ID":"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf","Type":"ContainerStarted","Data":"d05063308262ea235e7286d287e014a23517a43bc57a9b2a5855781befd8181c"}
Jul 23 15:58:27.108298 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.108289    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:27.108670 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.108304    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sk5fq"
Jul 23 15:58:27.116518 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.116498    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:27.118559 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.118543    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sk5fq"
Jul 23 15:58:27.230448 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.230426    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glv25\" (UniqueName: \"kubernetes.io/projected/e7165f08-e9a2-45bb-b4ac-996f6921c37d-kube-api-access-glv25\") pod \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\" (UID: \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\") "
Jul 23 15:58:27.230552 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.230468    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmcpt\" (UniqueName: \"kubernetes.io/projected/b389f46f-c69a-44a9-bc5a-6a70684a6dc9-kube-api-access-pmcpt\") pod \"b389f46f-c69a-44a9-bc5a-6a70684a6dc9\" (UID: \"b389f46f-c69a-44a9-bc5a-6a70684a6dc9\") "
Jul 23 15:58:27.230552 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.230509    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7165f08-e9a2-45bb-b4ac-996f6921c37d-tls-cert\") pod \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\" (UID: \"e7165f08-e9a2-45bb-b4ac-996f6921c37d\") "
Jul 23 15:58:27.232596 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.232571    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7165f08-e9a2-45bb-b4ac-996f6921c37d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e7165f08-e9a2-45bb-b4ac-996f6921c37d" (UID: "e7165f08-e9a2-45bb-b4ac-996f6921c37d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 15:58:27.232724 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.232620    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b389f46f-c69a-44a9-bc5a-6a70684a6dc9-kube-api-access-pmcpt" (OuterVolumeSpecName: "kube-api-access-pmcpt") pod "b389f46f-c69a-44a9-bc5a-6a70684a6dc9" (UID: "b389f46f-c69a-44a9-bc5a-6a70684a6dc9"). InnerVolumeSpecName "kube-api-access-pmcpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:58:27.232724 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.232645    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7165f08-e9a2-45bb-b4ac-996f6921c37d-kube-api-access-glv25" (OuterVolumeSpecName: "kube-api-access-glv25") pod "e7165f08-e9a2-45bb-b4ac-996f6921c37d" (UID: "e7165f08-e9a2-45bb-b4ac-996f6921c37d"). InnerVolumeSpecName "kube-api-access-glv25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:58:27.331490 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.331465    2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7165f08-e9a2-45bb-b4ac-996f6921c37d-tls-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:58:27.331490 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.331489    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glv25\" (UniqueName: \"kubernetes.io/projected/e7165f08-e9a2-45bb-b4ac-996f6921c37d-kube-api-access-glv25\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:58:27.331639 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:27.331500    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmcpt\" (UniqueName: \"kubernetes.io/projected/b389f46f-c69a-44a9-bc5a-6a70684a6dc9-kube-api-access-pmcpt\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:58:28.113691 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.113655    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-sk5fq"
Jul 23 15:58:28.113691 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.113663    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64bd574ffb-6qj68" event={"ID":"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf","Type":"ContainerStarted","Data":"21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c"}
Jul 23 15:58:28.114219 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.113854    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-xb7b6"
Jul 23 15:58:28.130070 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.130032    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64bd574ffb-6qj68" podStartSLOduration=1.703720833 podStartE2EDuration="2.130018467s" totalImagesPullingTime="426.297634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:58:28.129297753 +0000 UTC m=+595.754832229" watchObservedRunningTime="2026-07-23 15:58:28.130018467 +0000 UTC m=+595.755552942"
Jul 23 15:58:28.163117 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.163088    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cpk6b"]
Jul 23 15:58:28.163267 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.163243    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-cpk6b" podUID="9df83039-7476-4a24-81b9-7c122abf03f9" containerName="authorino" containerID="cri-o://d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b" gracePeriod=30
Jul 23 15:58:28.169350 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.169323    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-xb7b6"]
Jul 23 15:58:28.179211 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.179193    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-xb7b6"]
Jul 23 15:58:28.200786 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.200763    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sk5fq"]
Jul 23 15:58:28.206409 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.206388    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-sk5fq"]
Jul 23 15:58:28.401187 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.401165    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cpk6b"
Jul 23 15:58:28.541711 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.541682    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnmf8\" (UniqueName: \"kubernetes.io/projected/9df83039-7476-4a24-81b9-7c122abf03f9-kube-api-access-tnmf8\") pod \"9df83039-7476-4a24-81b9-7c122abf03f9\" (UID: \"9df83039-7476-4a24-81b9-7c122abf03f9\") "
Jul 23 15:58:28.543541 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.543517    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df83039-7476-4a24-81b9-7c122abf03f9-kube-api-access-tnmf8" (OuterVolumeSpecName: "kube-api-access-tnmf8") pod "9df83039-7476-4a24-81b9-7c122abf03f9" (UID: "9df83039-7476-4a24-81b9-7c122abf03f9"). InnerVolumeSpecName "kube-api-access-tnmf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 15:58:28.643034 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.642977    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnmf8\" (UniqueName: \"kubernetes.io/projected/9df83039-7476-4a24-81b9-7c122abf03f9-kube-api-access-tnmf8\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 15:58:28.809913 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.809887    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b389f46f-c69a-44a9-bc5a-6a70684a6dc9" path="/var/lib/kubelet/pods/b389f46f-c69a-44a9-bc5a-6a70684a6dc9/volumes"
Jul 23 15:58:28.810139 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:28.810127    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7165f08-e9a2-45bb-b4ac-996f6921c37d" path="/var/lib/kubelet/pods/e7165f08-e9a2-45bb-b4ac-996f6921c37d/volumes"
Jul 23 15:58:29.119434 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.119367    2555 generic.go:356] "Generic (PLEG): container finished" podID="9df83039-7476-4a24-81b9-7c122abf03f9" containerID="d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b" exitCode=0
Jul 23 15:58:29.119434 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.119403    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cpk6b"
Jul 23 15:58:29.120028 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.119447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cpk6b" event={"ID":"9df83039-7476-4a24-81b9-7c122abf03f9","Type":"ContainerDied","Data":"d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b"}
Jul 23 15:58:29.120028 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.119483    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cpk6b" event={"ID":"9df83039-7476-4a24-81b9-7c122abf03f9","Type":"ContainerDied","Data":"8ab95482dac5c6993dd6182517f317dd46f50aa8322c92cfc97e059f8b1ca22a"}
Jul 23 15:58:29.120028 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.119500    2555 scope.go:117] "RemoveContainer" containerID="d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b"
Jul 23 15:58:29.128026 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.128007    2555 scope.go:117] "RemoveContainer" containerID="d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b"
Jul 23 15:58:29.128291 ip-10-0-128-121 kubenswrapper[2555]: E0723 15:58:29.128271    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b\": container with ID starting with d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b not found: ID does not exist" containerID="d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b"
Jul 23 15:58:29.128355 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.128302    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b"} err="failed to get container status \"d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b\": rpc error: code = NotFound desc = could not find container \"d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b\": container with ID starting with d84f12e8b67b83ad5829f71ca69c87a43a2ea1c1f931172e64aaabf7659c751b not found: ID does not exist"
Jul 23 15:58:29.134455 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.134437    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cpk6b"]
Jul 23 15:58:29.139462 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:29.139441    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cpk6b"]
Jul 23 15:58:30.809896 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:30.809865    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df83039-7476-4a24-81b9-7c122abf03f9" path="/var/lib/kubelet/pods/9df83039-7476-4a24-81b9-7c122abf03f9/volumes"
Jul 23 15:58:32.722939 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:32.722909    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:58:32.723432 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:58:32.723417    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 15:59:10.183805 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:10.183769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 15:59:18.324251 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:18.324219    2555 generic.go:356] "Generic (PLEG): container finished" podID="1c022797-c084-4d04-ac20-05fdf9f9d87b" containerID="71f4cd9843cf1daeaa54f902a9debfb9f0c13cb322ee59f957c5f4847851d776" exitCode=0
Jul 23 15:59:18.324641 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:18.324293    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" event={"ID":"1c022797-c084-4d04-ac20-05fdf9f9d87b","Type":"ContainerDied","Data":"71f4cd9843cf1daeaa54f902a9debfb9f0c13cb322ee59f957c5f4847851d776"}
Jul 23 15:59:18.324641 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:18.324618    2555 scope.go:117] "RemoveContainer" containerID="71f4cd9843cf1daeaa54f902a9debfb9f0c13cb322ee59f957c5f4847851d776"
Jul 23 15:59:18.325497 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:18.325475    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:59:18.325550 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:18.325538    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:59:18.325625 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:18.325568    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 15:59:19.330131 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:19.330101    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" event={"ID":"1c022797-c084-4d04-ac20-05fdf9f9d87b","Type":"ContainerStarted","Data":"2523bf5184fe7de0f1c4e0adc21a9bf564da26fff41d7940ec6fc02ef1d32e9c"}
Jul 23 15:59:19.651979 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:19.651953    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:59:19.653133 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:19.653106    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rp4xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Jul 23 15:59:19.653641 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:19.653333    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" podUID="1c022797-c084-4d04-ac20-05fdf9f9d87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Jul 23 15:59:20.651310 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:20.651278    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rp4xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Jul 23 15:59:20.651659 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:20.651330    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" podUID="1c022797-c084-4d04-ac20-05fdf9f9d87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Jul 23 15:59:20.939906 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:20.939833    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:59:21.651493 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:21.651453    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rp4xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Jul 23 15:59:21.651904 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:21.651510    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" podUID="1c022797-c084-4d04-ac20-05fdf9f9d87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Jul 23 15:59:22.651284 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:22.651250    2555 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rp4xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Jul 23 15:59:22.651466 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:22.651304    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm" podUID="1c022797-c084-4d04-ac20-05fdf9f9d87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Jul 23 15:59:23.655710 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:23.655681    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:59:23.656772 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:23.656728    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rp4xm"
Jul 23 15:59:36.338181 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:36.338142    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 15:59:47.127425 ip-10-0-128-121 kubenswrapper[2555]: I0723 15:59:47.127391    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:00:07.927125 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:07.927092    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:00:39.302529 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.302452    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"]
Jul 23 16:00:39.303035 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.302812    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df83039-7476-4a24-81b9-7c122abf03f9" containerName="authorino"
Jul 23 16:00:39.303035 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.302825    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df83039-7476-4a24-81b9-7c122abf03f9" containerName="authorino"
Jul 23 16:00:39.303035 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.302891    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df83039-7476-4a24-81b9-7c122abf03f9" containerName="authorino"
Jul 23 16:00:39.305791 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.305774    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.310110 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.310088    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 23 16:00:39.310211 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.310088    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vlp5j\""
Jul 23 16:00:39.310211 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.310094    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 16:00:39.310384 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.310367    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 16:00:39.316270 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.316248    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"]
Jul 23 16:00:39.416422 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.416388    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.416614 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.416428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.416614 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.416451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.416614 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.416472    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.416788 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.416698    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e371a2f-7379-4784-9b61-efc9f58efba5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.416840 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.416804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98gx\" (UniqueName: \"kubernetes.io/projected/9e371a2f-7379-4784-9b61-efc9f58efba5-kube-api-access-g98gx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517207 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517172    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98gx\" (UniqueName: \"kubernetes.io/projected/9e371a2f-7379-4784-9b61-efc9f58efba5-kube-api-access-g98gx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517379 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517216    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517379 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517379 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517379 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517267    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517379 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e371a2f-7379-4784-9b61-efc9f58efba5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517634 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517607    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517708 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517686    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.517795 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.517724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.519438 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.519416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e371a2f-7379-4784-9b61-efc9f58efba5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.519843 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.519824    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e371a2f-7379-4784-9b61-efc9f58efba5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.525484 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.525466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98gx\" (UniqueName: \"kubernetes.io/projected/9e371a2f-7379-4784-9b61-efc9f58efba5-kube-api-access-g98gx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz\" (UID: \"9e371a2f-7379-4784-9b61-efc9f58efba5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.616605 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.616523    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:39.750832 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.750734    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"]
Jul 23 16:00:39.753122 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:00:39.753086    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e371a2f_7379_4784_9b61_efc9f58efba5.slice/crio-6550aa35bdbb43f6793dd0b79564935ef5931d184d3729c0276b470d81196bdd WatchSource:0}: Error finding container 6550aa35bdbb43f6793dd0b79564935ef5931d184d3729c0276b470d81196bdd: Status 404 returned error can't find the container with id 6550aa35bdbb43f6793dd0b79564935ef5931d184d3729c0276b470d81196bdd
Jul 23 16:00:39.754869 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:39.754849    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 16:00:40.679441 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:40.679405    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz" event={"ID":"9e371a2f-7379-4784-9b61-efc9f58efba5","Type":"ContainerStarted","Data":"6550aa35bdbb43f6793dd0b79564935ef5931d184d3729c0276b470d81196bdd"}
Jul 23 16:00:40.822353 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:40.822316    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:00:41.685341 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:41.685298    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz" event={"ID":"9e371a2f-7379-4784-9b61-efc9f58efba5","Type":"ContainerStarted","Data":"f41175bd141ee391479df6c6ab947931ab273d56a2755c979395bf6d7fb313af"}
Jul 23 16:00:41.708498 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:41.708451    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz" podStartSLOduration=1.427720057 podStartE2EDuration="2.708434901s" totalImagesPullingTime="1.280714844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:00:41.706305691 +0000 UTC m=+729.331840168" watchObservedRunningTime="2026-07-23 16:00:41.708434901 +0000 UTC m=+729.333969416"
Jul 23 16:00:49.616737 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:49.616702    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:49.617118 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:49.616804    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:49.629146 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:49.629124    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:49.734853 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:49.734827    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz"
Jul 23 16:00:58.140627 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.140586    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:00:58.518609 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.518527    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"]
Jul 23 16:00:58.522065 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.522045    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.524777 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.524742    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 23 16:00:58.532054 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.532032    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"]
Jul 23 16:00:58.575790 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.575744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.575980 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.575854    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.575980 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.575893    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk5l\" (UniqueName: \"kubernetes.io/projected/b275f199-979b-4429-beb3-3d1f7a55e332-kube-api-access-sbk5l\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.576079 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.575991    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.576079 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.576030    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.576079 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.576060    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b275f199-979b-4429-beb3-3d1f7a55e332-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.676606 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.676563    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.676803 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.676620    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b275f199-979b-4429-beb3-3d1f7a55e332-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.676803 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.676695    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.676803 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.676775    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.676991 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.676801    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk5l\" (UniqueName: \"kubernetes.io/projected/b275f199-979b-4429-beb3-3d1f7a55e332-kube-api-access-sbk5l\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.676991 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.676866    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.677097 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.677029    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.677169 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.677108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.677253 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.677226    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.678996 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.678972    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b275f199-979b-4429-beb3-3d1f7a55e332-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.679173 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.679155    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b275f199-979b-4429-beb3-3d1f7a55e332-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.685553 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.685528    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk5l\" (UniqueName: \"kubernetes.io/projected/b275f199-979b-4429-beb3-3d1f7a55e332-kube-api-access-sbk5l\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv\" (UID: \"b275f199-979b-4429-beb3-3d1f7a55e332\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:58.831562 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:58.831476    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:00:59.012345 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:59.012294    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"]
Jul 23 16:00:59.015176 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:00:59.015147    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb275f199_979b_4429_beb3_3d1f7a55e332.slice/crio-2f48e9179cf21b4ac1842d35ceafbecd22935ac3f41ec237a6a60bb58b754305 WatchSource:0}: Error finding container 2f48e9179cf21b4ac1842d35ceafbecd22935ac3f41ec237a6a60bb58b754305: Status 404 returned error can't find the container with id 2f48e9179cf21b4ac1842d35ceafbecd22935ac3f41ec237a6a60bb58b754305
Jul 23 16:00:59.771202 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:59.771162    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv" event={"ID":"b275f199-979b-4429-beb3-3d1f7a55e332","Type":"ContainerStarted","Data":"4230517a7924be196e58ec960f522f4f222324089ef8020ceeebb0deffc8eef4"}
Jul 23 16:00:59.771572 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:59.771208    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv" event={"ID":"b275f199-979b-4429-beb3-3d1f7a55e332","Type":"ContainerStarted","Data":"2f48e9179cf21b4ac1842d35ceafbecd22935ac3f41ec237a6a60bb58b754305"}
Jul 23 16:00:59.809375 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:00:59.809310    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv" podStartSLOduration=1.563812472 podStartE2EDuration="1.809293788s" totalImagesPullingTime="245.481316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:00:59.807553675 +0000 UTC m=+747.433088147" watchObservedRunningTime="2026-07-23 16:00:59.809293788 +0000 UTC m=+747.434828264"
Jul 23 16:01:08.832622 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:01:08.832587    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:01:08.833029 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:01:08.832637    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:01:08.844819 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:01:08.844796    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:01:09.615728 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:01:09.615695    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:01:09.832429 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:01:09.832401    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv"
Jul 23 16:02:10.693365 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.693328    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bc77ccfd9-vnwg6"]
Jul 23 16:02:10.696645 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.696625    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6"
Jul 23 16:02:10.706135 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.706112    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bc77ccfd9-vnwg6"]
Jul 23 16:02:10.750920 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.750887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgm2\" (UniqueName: \"kubernetes.io/projected/2af302f6-1d3b-4f87-ab06-1c86c307b325-kube-api-access-9zgm2\") pod \"authorino-6bc77ccfd9-vnwg6\" (UID: \"2af302f6-1d3b-4f87-ab06-1c86c307b325\") " pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6"
Jul 23 16:02:10.751033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.751001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2af302f6-1d3b-4f87-ab06-1c86c307b325-tls-cert\") pod \"authorino-6bc77ccfd9-vnwg6\" (UID: \"2af302f6-1d3b-4f87-ab06-1c86c307b325\") " pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6"
Jul 23 16:02:10.851535 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.851500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2af302f6-1d3b-4f87-ab06-1c86c307b325-tls-cert\") pod \"authorino-6bc77ccfd9-vnwg6\" (UID: \"2af302f6-1d3b-4f87-ab06-1c86c307b325\") " pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6"
Jul 23 16:02:10.851661 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.851561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgm2\" (UniqueName: \"kubernetes.io/projected/2af302f6-1d3b-4f87-ab06-1c86c307b325-kube-api-access-9zgm2\") pod \"authorino-6bc77ccfd9-vnwg6\" (UID: \"2af302f6-1d3b-4f87-ab06-1c86c307b325\") " pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6"
Jul 23 16:02:10.853866 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.853848    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2af302f6-1d3b-4f87-ab06-1c86c307b325-tls-cert\") pod \"authorino-6bc77ccfd9-vnwg6\" (UID: \"2af302f6-1d3b-4f87-ab06-1c86c307b325\") " pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6"
Jul 23 16:02:10.860152 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:10.860129    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgm2\" (UniqueName: \"kubernetes.io/projected/2af302f6-1d3b-4f87-ab06-1c86c307b325-kube-api-access-9zgm2\") pod \"authorino-6bc77ccfd9-vnwg6\" (UID: \"2af302f6-1d3b-4f87-ab06-1c86c307b325\") " pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6"
Jul 23 16:02:11.005635 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:11.005565    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6"
Jul 23 16:02:11.133137 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:11.133112    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bc77ccfd9-vnwg6"]
Jul 23 16:02:11.135361 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:02:11.135327    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2af302f6_1d3b_4f87_ab06_1c86c307b325.slice/crio-a7fd0ebffb03cfab5636af435f5c6efdc6c99418b00766b3f63007256a5de8ca WatchSource:0}: Error finding container a7fd0ebffb03cfab5636af435f5c6efdc6c99418b00766b3f63007256a5de8ca: Status 404 returned error can't find the container with id a7fd0ebffb03cfab5636af435f5c6efdc6c99418b00766b3f63007256a5de8ca
Jul 23 16:02:12.086316 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.086222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6" event={"ID":"2af302f6-1d3b-4f87-ab06-1c86c307b325","Type":"ContainerStarted","Data":"187bf1285c35f3f1cfed8ac0df19ef787c26e28023c65020ddffda589abf31c3"}
Jul 23 16:02:12.086316 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.086285    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6" event={"ID":"2af302f6-1d3b-4f87-ab06-1c86c307b325","Type":"ContainerStarted","Data":"a7fd0ebffb03cfab5636af435f5c6efdc6c99418b00766b3f63007256a5de8ca"}
Jul 23 16:02:12.122479 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.122413    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6bc77ccfd9-vnwg6" podStartSLOduration=1.5941963609999998 podStartE2EDuration="2.122390749s" totalImagesPullingTime="528.194388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:02:12.121056052 +0000 UTC m=+819.746590530" watchObservedRunningTime="2026-07-23 16:02:12.122390749 +0000 UTC m=+819.747925227"
Jul 23 16:02:12.172650 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.172612    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64bd574ffb-6qj68"]
Jul 23 16:02:12.173429 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.173373    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-64bd574ffb-6qj68" podUID="807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf" containerName="authorino" containerID="cri-o://21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c" gracePeriod=30
Jul 23 16:02:12.432986 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.432954    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 16:02:12.466768 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.466716    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g54gn\" (UniqueName: \"kubernetes.io/projected/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-kube-api-access-g54gn\") pod \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\" (UID: \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\") "
Jul 23 16:02:12.466940 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.466810    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-tls-cert\") pod \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\" (UID: \"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf\") "
Jul 23 16:02:12.468883 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.468853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-kube-api-access-g54gn" (OuterVolumeSpecName: "kube-api-access-g54gn") pod "807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf" (UID: "807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf"). InnerVolumeSpecName "kube-api-access-g54gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:02:12.481985 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.478686    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf" (UID: "807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 16:02:12.567993 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.567959    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g54gn\" (UniqueName: \"kubernetes.io/projected/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-kube-api-access-g54gn\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:02:12.567993 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:12.567987    2555 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf-tls-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:02:13.092628 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.092544    2555 generic.go:356] "Generic (PLEG): container finished" podID="807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf" containerID="21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c" exitCode=0
Jul 23 16:02:13.092628 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.092601    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64bd574ffb-6qj68"
Jul 23 16:02:13.093117 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.092631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64bd574ffb-6qj68" event={"ID":"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf","Type":"ContainerDied","Data":"21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c"}
Jul 23 16:02:13.093117 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.092671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64bd574ffb-6qj68" event={"ID":"807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf","Type":"ContainerDied","Data":"d05063308262ea235e7286d287e014a23517a43bc57a9b2a5855781befd8181c"}
Jul 23 16:02:13.093117 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.092690    2555 scope.go:117] "RemoveContainer" containerID="21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c"
Jul 23 16:02:13.101048 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.101025    2555 scope.go:117] "RemoveContainer" containerID="21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c"
Jul 23 16:02:13.101389 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:02:13.101349    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c\": container with ID starting with 21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c not found: ID does not exist" containerID="21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c"
Jul 23 16:02:13.101484 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.101389    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c"} err="failed to get container status \"21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c\": rpc error: code = NotFound desc = could not find container \"21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c\": container with ID starting with 21b26ec8658abb5f54b2d38a306e05ad1d7b67e987b88388fd758253a3be4d8c not found: ID does not exist"
Jul 23 16:02:13.115185 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.115162    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64bd574ffb-6qj68"]
Jul 23 16:02:13.119022 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:13.119003    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64bd574ffb-6qj68"]
Jul 23 16:02:14.809716 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:14.809680    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf" path="/var/lib/kubelet/pods/807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf/volumes"
Jul 23 16:02:40.323140 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:40.323102    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:02:51.418257 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:02:51.418212    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:03:02.014572 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:03:02.014534    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:03:32.751260 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:03:32.751177    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:03:32.751806 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:03:32.751392    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:04:47.815348 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:04:47.815311    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:05:27.330901 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:05:27.330866    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:05:37.610298 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:05:37.610259    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:05:46.432554 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:05:46.432517    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:05:58.112595 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:05:58.112560    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:06:09.313002 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:06:09.312963    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:06:18.914308 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:06:18.914270    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:06:30.250084 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:06:30.250010    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:06:40.207800 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:06:40.207762    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:07:05.553537 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:07:05.553503    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:07:17.457001 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:07:17.456967    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:07:40.127938 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:07:40.127907    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:07:52.811852 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:07:52.811822    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:08:08.740013 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:08:08.739976    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:08:18.115580 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:08:18.115535    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:08:27.913336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:08:27.913306    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:08:32.775342 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:08:32.775312    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:08:32.778183 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:08:32.778159    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:08:37.536149 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:08:37.536116    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:08:46.913669 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:08:46.913635    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:10:36.519357 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:10:36.519323    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:10:47.224368 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:10:47.224328    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:10:57.909413 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:10:57.909323    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:11:09.056414 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:11:09.056382    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:11:20.210720 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:11:20.210671    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:11:31.111657 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:11:31.111619    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:11:39.525131 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:11:39.525097    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:11:50.655168 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:11:50.655131    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:12:01.318246 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:01.318213    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:12:10.734241 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:10.734207    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:12:22.311502 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:22.311469    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:12:46.006557 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:46.006478    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:12:51.341744 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.341713    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"]
Jul 23 16:12:51.342108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.342053    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf" containerName="authorino"
Jul 23 16:12:51.342108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.342066    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf" containerName="authorino"
Jul 23 16:12:51.342183 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.342118    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="807971a2-a2e8-43d6-bb8f-26a4f2b6b2bf" containerName="authorino"
Jul 23 16:12:51.345048 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.345033    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.348194 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.348174    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bbbeb353-kserve-self-signed-certs\""
Jul 23 16:12:51.359854 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.359832    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"]
Jul 23 16:12:51.480386 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.480358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-home\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.480501 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.480396    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tmp-dir\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.480501 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.480416    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-model-cache\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.480578 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.480496    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-dshm\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.480578 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.480545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-kube-api-access-4hm6f\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.480642 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.480584    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tls-certs\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.581738 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.581705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-home\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.581903 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.581770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tmp-dir\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.581903 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.581786    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-model-cache\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.581903 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.581814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-dshm\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.581903 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.581849    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-kube-api-access-4hm6f\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.581903 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.581878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tls-certs\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.582188 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.582158    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-home\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.582188 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.582176    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tmp-dir\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.582280 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.582213    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-model-cache\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.584223 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.584191    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-dshm\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.584385 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.584368    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tls-certs\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.590788 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.590727    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-kube-api-access-4hm6f\") pod \"e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") " pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.654491 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.654468    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:12:51.782565 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.782539    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"]
Jul 23 16:12:51.784880 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:12:51.784851    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eab32a0_2478_44ef_a6b0_1e4ac94c2dcd.slice/crio-8588c2350d3beb36a0833fce72332fc51c49492b36bc5c0789576e328cfffc4d WatchSource:0}: Error finding container 8588c2350d3beb36a0833fce72332fc51c49492b36bc5c0789576e328cfffc4d: Status 404 returned error can't find the container with id 8588c2350d3beb36a0833fce72332fc51c49492b36bc5c0789576e328cfffc4d
Jul 23 16:12:51.911048 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.910969    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt" event={"ID":"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd","Type":"ContainerStarted","Data":"ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e"}
Jul 23 16:12:51.911048 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.911023    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt" event={"ID":"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd","Type":"ContainerStarted","Data":"8588c2350d3beb36a0833fce72332fc51c49492b36bc5c0789576e328cfffc4d"}
Jul 23 16:12:51.929977 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:51.929932    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt" podStartSLOduration=0.92991631 podStartE2EDuration="929.91631ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:12:51.928409902 +0000 UTC m=+1459.553944379" watchObservedRunningTime="2026-07-23 16:12:51.92991631 +0000 UTC m=+1459.555450787"
Jul 23 16:12:54.220523 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:12:54.220490    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:13:01.655290 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:01.655260    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:13:01.655628 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:01.655300    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:13:01.667838 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:01.667816    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:13:01.967856 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:01.967770    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:13:10.509805 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:10.509768    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:13:32.800693 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:32.800664    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:13:32.803928 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:32.803902    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:13:34.617895 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:34.617832    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"]
Jul 23 16:13:38.322732 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:38.322692    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:13:49.638189 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.638127    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt" podUID="4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" containerName="main" containerID="cri-o://ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e" gracePeriod=45
Jul 23 16:13:49.888793 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.888727    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:13:49.937892 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.937870    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-model-cache\") pod \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") "
Jul 23 16:13:49.938033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.937908    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-dshm\") pod \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") "
Jul 23 16:13:49.938033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.937971    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tmp-dir\") pod \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") "
Jul 23 16:13:49.938033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.938001    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tls-certs\") pod \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") "
Jul 23 16:13:49.938190 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.938079    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-kube-api-access-4hm6f\") pod \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") "
Jul 23 16:13:49.938190 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.938114    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-home\") pod \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\" (UID: \"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd\") "
Jul 23 16:13:49.938190 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.938112    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-model-cache" (OuterVolumeSpecName: "model-cache") pod "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" (UID: "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:13:49.938332 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.938185    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" (UID: "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:13:49.938420 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.938397    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-home" (OuterVolumeSpecName: "home") pod "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" (UID: "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:13:49.938420 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.938414    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:13:49.938501 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.938432    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:13:49.940090 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.940064    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-dshm" (OuterVolumeSpecName: "dshm") pod "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" (UID: "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:13:49.940184 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.940145    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" (UID: "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 16:13:49.940253 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:49.940234    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-kube-api-access-4hm6f" (OuterVolumeSpecName: "kube-api-access-4hm6f") pod "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" (UID: "4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd"). InnerVolumeSpecName "kube-api-access-4hm6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:13:50.039521 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.039496    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:13:50.039521 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.039519    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:13:50.039660 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.039527    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:13:50.039660 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.039537    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hm6f\" (UniqueName: \"kubernetes.io/projected/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd-kube-api-access-4hm6f\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:13:50.165891 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.165820    2555 generic.go:356] "Generic (PLEG): container finished" podID="4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" containerID="ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e" exitCode=0
Jul 23 16:13:50.165891 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.165860    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"
Jul 23 16:13:50.166038 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.165904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt" event={"ID":"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd","Type":"ContainerDied","Data":"ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e"}
Jul 23 16:13:50.166038 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.165943    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt" event={"ID":"4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd","Type":"ContainerDied","Data":"8588c2350d3beb36a0833fce72332fc51c49492b36bc5c0789576e328cfffc4d"}
Jul 23 16:13:50.166038 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.165960    2555 scope.go:117] "RemoveContainer" containerID="ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e"
Jul 23 16:13:50.174467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.174451    2555 scope.go:117] "RemoveContainer" containerID="ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e"
Jul 23 16:13:50.174715 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:13:50.174689    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e\": container with ID starting with ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e not found: ID does not exist" containerID="ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e"
Jul 23 16:13:50.174782 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.174722    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e"} err="failed to get container status \"ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e\": rpc error: code = NotFound desc = could not find container \"ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e\": container with ID starting with ef0ab214b8af0869e9da627af5b73ba42e88f13eb0a0fecc0db118572a71850e not found: ID does not exist"
Jul 23 16:13:50.185825 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.185806    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"]
Jul 23 16:13:50.190052 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.190030    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bbbeb353-kserve-6975f9855-krhzt"]
Jul 23 16:13:50.809996 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.809965    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" path="/var/lib/kubelet/pods/4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd/volumes"
Jul 23 16:13:50.820571 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:13:50.820544    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:14:01.123867 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:14:01.123782    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:14:14.226661 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:14:14.226629    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:14:26.107794 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:14:26.107760    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:14:35.810330 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:14:35.810297    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:14:44.121688 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:14:44.121652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:14:54.922477 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:14:54.922445    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:15:04.216800 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:15:04.216763    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:15:17.714776 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:15:17.714707    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:15:27.310129 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:15:27.310048    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:15:37.425143 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:15:37.425100    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:15:49.835121 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:15:49.835086    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:16:03.123476 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:16:03.123445    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:16:11.408982 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:16:11.408950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:16:23.028265 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:16:23.028232    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:16:34.929069 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:16:34.929030    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:16:48.021387 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:16:48.021352    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:16:57.311402 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:16:57.311314    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:17:19.521472 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:19.521435    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:17:25.158193 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.158158    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"]
Jul 23 16:17:25.158692 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.158664    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" containerName="main"
Jul 23 16:17:25.158791 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.158695    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" containerName="main"
Jul 23 16:17:25.158849 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.158799    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eab32a0-2478-44ef-a6b0-1e4ac94c2dcd" containerName="main"
Jul 23 16:17:25.161936 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.161913    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.164462 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.164439    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8e175082-openshift-default-dockercfg-kwcbc\""
Jul 23 16:17:25.181878 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.181855    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"]
Jul 23 16:17:25.209360 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-token\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.209495 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad861b4-1c59-43b9-a212-13fe15c2ff21-istiod-ca-cert\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.209495 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209439    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrx6r\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-kube-api-access-xrx6r\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.209495 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209471    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-socket\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.209602 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-envoy\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.209602 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209573    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-certs\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.209602 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209598    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-podinfo\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.209707 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209621    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-data\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.209707 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.209640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-credential-socket\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311015 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.310976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-data\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311015 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-credential-socket\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311274 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311048    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-token\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311274 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad861b4-1c59-43b9-a212-13fe15c2ff21-istiod-ca-cert\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311274 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311161    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrx6r\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-kube-api-access-xrx6r\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311274 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-socket\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311274 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311227    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-envoy\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311274 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311268    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-certs\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311573 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-podinfo\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311573 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311437    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-data\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311573 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311512    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-credential-socket\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311727 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311688    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-certs\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311823 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311784    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-socket\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.311919 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.311894    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad861b4-1c59-43b9-a212-13fe15c2ff21-istiod-ca-cert\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.313578 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.313554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-envoy\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.313873 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.313855    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-podinfo\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.326857 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.326836    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-token\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.327163 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.327141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrx6r\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-kube-api-access-xrx6r\") pod \"e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") " pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.472537 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.472432    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:25.613334 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.613308    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"]
Jul 23 16:17:25.615710 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:17:25.615679    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad861b4_1c59_43b9_a212_13fe15c2ff21.slice/crio-bf3fc6f0aef8fbf7691a2fc13290356fc7acbe8ccd1c6fc77af89340cc2359b7 WatchSource:0}: Error finding container bf3fc6f0aef8fbf7691a2fc13290356fc7acbe8ccd1c6fc77af89340cc2359b7: Status 404 returned error can't find the container with id bf3fc6f0aef8fbf7691a2fc13290356fc7acbe8ccd1c6fc77af89340cc2359b7
Jul 23 16:17:25.623785 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.623712    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:17:25.623886 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.623855    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:17:25.623931 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:25.623915    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:17:26.104093 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:26.104058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" event={"ID":"cad861b4-1c59-43b9-a212-13fe15c2ff21","Type":"ContainerStarted","Data":"dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28"}
Jul 23 16:17:26.104262 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:26.104117    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" event={"ID":"cad861b4-1c59-43b9-a212-13fe15c2ff21","Type":"ContainerStarted","Data":"bf3fc6f0aef8fbf7691a2fc13290356fc7acbe8ccd1c6fc77af89340cc2359b7"}
Jul 23 16:17:26.124683 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:26.124626    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" podStartSLOduration=1.124610091 podStartE2EDuration="1.124610091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:17:26.124227521 +0000 UTC m=+1733.749761998" watchObservedRunningTime="2026-07-23 16:17:26.124610091 +0000 UTC m=+1733.750144568"
Jul 23 16:17:26.472931 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:26.472892    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:26.477880 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:26.477853    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:27.108893 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:27.108867    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:27.109710 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:27.109694    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:33.218386 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:33.218350    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:17:41.174598 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:41.174563    2555 generic.go:356] "Generic (PLEG): container finished" podID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerID="dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28" exitCode=0
Jul 23 16:17:41.175051 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:41.174612    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" event={"ID":"cad861b4-1c59-43b9-a212-13fe15c2ff21","Type":"ContainerDied","Data":"dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28"}
Jul 23 16:17:41.175051 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:41.174991    2555 scope.go:117] "RemoveContainer" containerID="dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28"
Jul 23 16:17:41.176001 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:41.175970    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:17:41.176075 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:41.176044    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:17:41.176110 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:41.176077    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:17:42.109106 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:42.109070    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:42.180816 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:42.180780    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" event={"ID":"cad861b4-1c59-43b9-a212-13fe15c2ff21","Type":"ContainerStarted","Data":"e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388"}
Jul 23 16:17:42.473025 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:42.472986    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:42.474200 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:42.474178    2555 patch_prober.go:28] interesting pod/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 23 16:17:42.474308 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:42.474233    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 23 16:17:43.473536 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:43.473498    2555 patch_prober.go:28] interesting pod/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 23 16:17:43.474993 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:43.474960    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 23 16:17:44.473377 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:44.473337    2555 patch_prober.go:28] interesting pod/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 23 16:17:44.473593 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:44.473408    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 23 16:17:45.472950 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:45.472911    2555 patch_prober.go:28] interesting pod/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 23 16:17:45.473262 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:45.473230    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 23 16:17:46.473531 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:46.473487    2555 patch_prober.go:28] interesting pod/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 23 16:17:46.474212 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:46.474177    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 23 16:17:47.477206 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:47.477167    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:47.477582 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:47.477464    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:17:47.478227 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:17:47.478212    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:18:06.714454 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:06.714413    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:18:08.450212 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:08.450174    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"]
Jul 23 16:18:08.451030 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:08.450997    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy" containerID="cri-o://e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388" gracePeriod=30
Jul 23 16:18:09.802185 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.802153    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"]
Jul 23 16:18:09.805790 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.805769    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.809987 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.809969    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8f4ec811-gw-openshift-default-dockercfg-b8zfw\""
Jul 23 16:18:09.819986 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.819960    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-certs\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.820103 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.820000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjw2m\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-kube-api-access-bjw2m\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.820103 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.820039    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-data\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.820185 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.820125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298385fa-8136-472f-b285-962019f92ae9-istiod-ca-cert\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.820185 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.820158    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-socket\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.820185 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.820176    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298385fa-8136-472f-b285-962019f92ae9-istio-podinfo\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.820295 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.820226    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-envoy\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.820295 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.820259    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-istio-token\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.820367 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.820293    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-credential-socket\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.826636 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.826608    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"]
Jul 23 16:18:09.920935 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.920894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298385fa-8136-472f-b285-962019f92ae9-istiod-ca-cert\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921199 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921175    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-socket\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921330 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921207    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298385fa-8136-472f-b285-962019f92ae9-istio-podinfo\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921330 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921281    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-envoy\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921330 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-istio-token\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921571 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-credential-socket\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921571 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921421    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-certs\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921571 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921454    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjw2m\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-kube-api-access-bjw2m\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921571 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-data\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.921780 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.921718    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298385fa-8136-472f-b285-962019f92ae9-istiod-ca-cert\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.922190 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.922013    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-credential-socket\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.922190 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.922125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-data\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.922190 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.922126    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-certs\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.922387 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.922274    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-socket\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.926635 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.924590    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-envoy\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.926635 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.924924    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298385fa-8136-472f-b285-962019f92ae9-istio-podinfo\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.929987 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.929964    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-istio-token\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:09.930511 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:09.930493    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjw2m\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-kube-api-access-bjw2m\") pod \"e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") " pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:10.116376 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:10.116288    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:10.256986 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:10.256950    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"]
Jul 23 16:18:10.260358 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:18:10.260328    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298385fa_8136_472f_b285_962019f92ae9.slice/crio-b4c037a69ebf719ca8aa1f87655dd2f28567dba77af1b7dd5f3236b45b63f0a2 WatchSource:0}: Error finding container b4c037a69ebf719ca8aa1f87655dd2f28567dba77af1b7dd5f3236b45b63f0a2: Status 404 returned error can't find the container with id b4c037a69ebf719ca8aa1f87655dd2f28567dba77af1b7dd5f3236b45b63f0a2
Jul 23 16:18:10.262553 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:10.262521    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:10.262628 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:10.262587    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:10.262628 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:10.262624    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:10.304615 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:10.304583    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" event={"ID":"298385fa-8136-472f-b285-962019f92ae9","Type":"ContainerStarted","Data":"b4c037a69ebf719ca8aa1f87655dd2f28567dba77af1b7dd5f3236b45b63f0a2"}
Jul 23 16:18:11.310634 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:11.310592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" event={"ID":"298385fa-8136-472f-b285-962019f92ae9","Type":"ContainerStarted","Data":"4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9"}
Jul 23 16:18:11.333711 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:11.333654    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" podStartSLOduration=2.333638128 podStartE2EDuration="2.333638128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:18:11.331002616 +0000 UTC m=+1778.956537092" watchObservedRunningTime="2026-07-23 16:18:11.333638128 +0000 UTC m=+1778.959172582"
Jul 23 16:18:12.116605 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:12.116563    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:12.121761 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:12.121727    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:12.315616 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:12.315586    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:12.316509 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:12.316488    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:13.711964 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.711930    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:18:13.755990 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.755906    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-data\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.755990 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.755947    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-certs\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.755990 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.755989    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrx6r\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-kube-api-access-xrx6r\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.756275 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756030    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-podinfo\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.756275 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756061    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-credential-socket\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.756386 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756302    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:13.756386 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756323    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-token\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.756386 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756362    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-envoy\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.756593 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756395    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-socket\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.756593 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756471    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad861b4-1c59-43b9-a212-13fe15c2ff21-istiod-ca-cert\") pod \"cad861b4-1c59-43b9-a212-13fe15c2ff21\" (UID: \"cad861b4-1c59-43b9-a212-13fe15c2ff21\") "
Jul 23 16:18:13.756869 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756784    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-data" (OuterVolumeSpecName: "istio-data") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:13.757027 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.756803    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:13.757119 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.757061    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:13.757208 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.757173    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:13.757364 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.757338    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad861b4-1c59-43b9-a212-13fe15c2ff21-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:18:13.758531 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.758490    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 16:18:13.758961 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.758937    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:13.758961 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.758948    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-kube-api-access-xrx6r" (OuterVolumeSpecName: "kube-api-access-xrx6r") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "kube-api-access-xrx6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:18:13.759074 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.758964    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-token" (OuterVolumeSpecName: "istio-token") pod "cad861b4-1c59-43b9-a212-13fe15c2ff21" (UID: "cad861b4-1c59-43b9-a212-13fe15c2ff21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:18:13.858033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.857997    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrx6r\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-kube-api-access-xrx6r\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:13.858033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.858025    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-podinfo\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:13.858033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.858036    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-credential-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:13.858033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.858044    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:13.858289 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.858053    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-envoy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:13.858289 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.858061    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-workload-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:13.858289 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.858069    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad861b4-1c59-43b9-a212-13fe15c2ff21-istiod-ca-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:13.858289 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:13.858077    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad861b4-1c59-43b9-a212-13fe15c2ff21-istio-data\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:14.326284 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.326247    2555 generic.go:356] "Generic (PLEG): container finished" podID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerID="e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388" exitCode=0
Jul 23 16:18:14.326456 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.326292    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"
Jul 23 16:18:14.326456 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.326327    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" event={"ID":"cad861b4-1c59-43b9-a212-13fe15c2ff21","Type":"ContainerDied","Data":"e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388"}
Jul 23 16:18:14.326456 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.326379    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh" event={"ID":"cad861b4-1c59-43b9-a212-13fe15c2ff21","Type":"ContainerDied","Data":"bf3fc6f0aef8fbf7691a2fc13290356fc7acbe8ccd1c6fc77af89340cc2359b7"}
Jul 23 16:18:14.326456 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.326403    2555 scope.go:117] "RemoveContainer" containerID="e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388"
Jul 23 16:18:14.335366 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.335350    2555 scope.go:117] "RemoveContainer" containerID="dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28"
Jul 23 16:18:14.342854 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.342833    2555 scope.go:117] "RemoveContainer" containerID="e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388"
Jul 23 16:18:14.343105 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:18:14.343088    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388\": container with ID starting with e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388 not found: ID does not exist" containerID="e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388"
Jul 23 16:18:14.343150 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.343113    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388"} err="failed to get container status \"e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388\": rpc error: code = NotFound desc = could not find container \"e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388\": container with ID starting with e50da94d4fdd18b370c834d6861752bd17f2f6c4bcdab4c1dd5ab297271ed388 not found: ID does not exist"
Jul 23 16:18:14.343150 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.343128    2555 scope.go:117] "RemoveContainer" containerID="dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28"
Jul 23 16:18:14.343331 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:18:14.343315    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28\": container with ID starting with dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28 not found: ID does not exist" containerID="dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28"
Jul 23 16:18:14.343366 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.343336    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28"} err="failed to get container status \"dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28\": rpc error: code = NotFound desc = could not find container \"dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28\": container with ID starting with dd3908ea4f2034cc3b52012cc086951a8efb7716ecf17a112efc345677601f28 not found: ID does not exist"
Jul 23 16:18:14.347369 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.347345    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"]
Jul 23 16:18:14.351508 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.351488    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8e175082-openshift-default-65cbdd9759-khxwh"]
Jul 23 16:18:14.810102 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:14.810066    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" path="/var/lib/kubelet/pods/cad861b4-1c59-43b9-a212-13fe15c2ff21/volumes"
Jul 23 16:18:19.506343 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:19.506307    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:18:27.317500 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:27.317410    2555 patch_prober.go:28] interesting pod/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 23 16:18:27.317500 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:27.317474    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 23 16:18:27.406212 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:27.406174    2555 generic.go:356] "Generic (PLEG): container finished" podID="298385fa-8136-472f-b285-962019f92ae9" containerID="4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9" exitCode=0
Jul 23 16:18:27.406378 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:27.406219    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" event={"ID":"298385fa-8136-472f-b285-962019f92ae9","Type":"ContainerDied","Data":"4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9"}
Jul 23 16:18:27.406616 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:27.406601    2555 scope.go:117] "RemoveContainer" containerID="4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9"
Jul 23 16:18:27.407594 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:27.407564    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:27.407651 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:27.407632    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:27.407694 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:27.407665    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:28.412181 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:28.412148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" event={"ID":"298385fa-8136-472f-b285-962019f92ae9","Type":"ContainerStarted","Data":"7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19"}
Jul 23 16:18:29.116831 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:29.116788    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:29.117069 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:29.117049    2555 patch_prober.go:28] interesting pod/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 23 16:18:29.117116 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:29.117094    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 23 16:18:30.117484 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:30.117443    2555 patch_prober.go:28] interesting pod/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 23 16:18:30.117926 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:30.117502    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 23 16:18:31.117149 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:31.117110    2555 patch_prober.go:28] interesting pod/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 23 16:18:31.117359 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:31.117168    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 23 16:18:32.117026 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:32.116986    2555 patch_prober.go:28] interesting pod/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 23 16:18:32.117443 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:32.117045    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 23 16:18:32.827479 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:32.827452    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:18:32.831440 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:32.831411    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:18:33.121080 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:33.121004    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:33.121416 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:33.121331    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:33.122074 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:33.122059    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:51.520412 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:51.520375    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:18:53.216056 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:53.216022    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"]
Jul 23 16:18:53.216659 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:53.216284    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy" containerID="cri-o://7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19" gracePeriod=30
Jul 23 16:18:54.061769 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.057205    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"]
Jul 23 16:18:54.061769 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.058055    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy"
Jul 23 16:18:54.061769 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.058075    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy"
Jul 23 16:18:54.061769 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.058287    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy"
Jul 23 16:18:54.061769 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.058300    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy"
Jul 23 16:18:54.064176 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.064147    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.066873 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.066844    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e475fc66-openshift-default-dockercfg-2mrdj\""
Jul 23 16:18:54.073555 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.073529    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"]
Jul 23 16:18:54.201506 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201475    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istiod-ca-cert\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.201506 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201511    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzt6r\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-kube-api-access-tzt6r\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.201743 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-envoy\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.201743 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-data\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.201743 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-socket\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.201743 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201717    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-credential-socket\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.201743 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201735    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-podinfo\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.202023 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201811    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-certs\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.202023 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.201833    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-token\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.302982 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.302940    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istiod-ca-cert\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.302982 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.302988    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzt6r\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-kube-api-access-tzt6r\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-envoy\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-data\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303112    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-socket\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303193    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-credential-socket\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-podinfo\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-certs\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-token\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303839 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-data\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303839 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-credential-socket\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303839 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303785    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-socket\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303985 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303923    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-certs\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.303985 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.303934    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istiod-ca-cert\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.305313 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.305281    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-envoy\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.305675 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.305651    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-podinfo\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.310956 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.310933    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-token\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.311221 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.311194    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzt6r\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-kube-api-access-tzt6r\") pod \"e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") " pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.376174 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.376108    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:54.512272 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.512247    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"]
Jul 23 16:18:54.514536 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:18:54.514501    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4209e0_f0b3_4da9_8cfa_1b12e78b5f81.slice/crio-8fcf6725e57cf6baa83692a063091af21cec2a266e8683b8edb4400f5fa81b55 WatchSource:0}: Error finding container 8fcf6725e57cf6baa83692a063091af21cec2a266e8683b8edb4400f5fa81b55: Status 404 returned error can't find the container with id 8fcf6725e57cf6baa83692a063091af21cec2a266e8683b8edb4400f5fa81b55
Jul 23 16:18:54.517116 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.517078    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:54.517232 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.517173    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:54.517232 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.517224    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:18:54.536146 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:54.536113    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" event={"ID":"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81","Type":"ContainerStarted","Data":"8fcf6725e57cf6baa83692a063091af21cec2a266e8683b8edb4400f5fa81b55"}
Jul 23 16:18:55.542107 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:55.542072    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" event={"ID":"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81","Type":"ContainerStarted","Data":"b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e"}
Jul 23 16:18:55.564655 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:55.564605    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" podStartSLOduration=1.564589324 podStartE2EDuration="1.564589324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:18:55.564190754 +0000 UTC m=+1823.189725231" watchObservedRunningTime="2026-07-23 16:18:55.564589324 +0000 UTC m=+1823.190123799"
Jul 23 16:18:56.376789 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:56.376731    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:56.381902 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:56.381879    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:56.546474 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:56.546440    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:56.547492 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:56.547463    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:18:58.477186 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.477133    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:58.557472 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.557446    2555 generic.go:356] "Generic (PLEG): container finished" podID="298385fa-8136-472f-b285-962019f92ae9" containerID="7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19" exitCode=0
Jul 23 16:18:58.557601 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.557494    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"
Jul 23 16:18:58.557601 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.557521    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" event={"ID":"298385fa-8136-472f-b285-962019f92ae9","Type":"ContainerDied","Data":"7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19"}
Jul 23 16:18:58.557601 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.557568    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h" event={"ID":"298385fa-8136-472f-b285-962019f92ae9","Type":"ContainerDied","Data":"b4c037a69ebf719ca8aa1f87655dd2f28567dba77af1b7dd5f3236b45b63f0a2"}
Jul 23 16:18:58.557601 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.557590    2555 scope.go:117] "RemoveContainer" containerID="7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19"
Jul 23 16:18:58.565488 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.565471    2555 scope.go:117] "RemoveContainer" containerID="4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9"
Jul 23 16:18:58.572772 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.572728    2555 scope.go:117] "RemoveContainer" containerID="7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19"
Jul 23 16:18:58.573078 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:18:58.573061    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19\": container with ID starting with 7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19 not found: ID does not exist" containerID="7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19"
Jul 23 16:18:58.573145 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.573085    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19"} err="failed to get container status \"7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19\": rpc error: code = NotFound desc = could not find container \"7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19\": container with ID starting with 7240794f300d1fd50cbaca8bb5428979e63071d177c01d6e128cb35ecd5aad19 not found: ID does not exist"
Jul 23 16:18:58.573145 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.573102    2555 scope.go:117] "RemoveContainer" containerID="4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9"
Jul 23 16:18:58.573362 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:18:58.573344    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9\": container with ID starting with 4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9 not found: ID does not exist" containerID="4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9"
Jul 23 16:18:58.573415 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.573370    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9"} err="failed to get container status \"4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9\": rpc error: code = NotFound desc = could not find container \"4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9\": container with ID starting with 4bb92131dcd0d7170639fc7bd315115545daaca3a03f0d76fa5af0f8401e8bf9 not found: ID does not exist"
Jul 23 16:18:58.643737 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643714    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-socket\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.643850 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643742    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298385fa-8136-472f-b285-962019f92ae9-istio-podinfo\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.643850 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643785    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-istio-token\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.643850 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643818    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-envoy\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.643850 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643844    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-credential-socket\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.644000 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643879    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298385fa-8136-472f-b285-962019f92ae9-istiod-ca-cert\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.644000 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643936    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-data\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.644000 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643980    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:58.644154 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.643987    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-certs\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.644154 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.644123    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjw2m\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-kube-api-access-bjw2m\") pod \"298385fa-8136-472f-b285-962019f92ae9\" (UID: \"298385fa-8136-472f-b285-962019f92ae9\") "
Jul 23 16:18:58.644255 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.644147    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:58.644255 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.644197    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:58.644469 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.644446    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.644551 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.644472    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-credential-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.644551 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.644488    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-workload-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.644861 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.644837    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298385fa-8136-472f-b285-962019f92ae9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:18:58.645111 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.645088    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-data" (OuterVolumeSpecName: "istio-data") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:58.645944 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.645923    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/298385fa-8136-472f-b285-962019f92ae9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 16:18:58.646160 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.646128    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-kube-api-access-bjw2m" (OuterVolumeSpecName: "kube-api-access-bjw2m") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "kube-api-access-bjw2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:18:58.646339 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.646321    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-istio-token" (OuterVolumeSpecName: "istio-token") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:18:58.646339 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.646325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "298385fa-8136-472f-b285-962019f92ae9" (UID: "298385fa-8136-472f-b285-962019f92ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:18:58.745528 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.745466    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjw2m\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-kube-api-access-bjw2m\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.745528 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.745490    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/298385fa-8136-472f-b285-962019f92ae9-istio-podinfo\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.745528 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.745502    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/298385fa-8136-472f-b285-962019f92ae9-istio-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.745528 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.745514    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-envoy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.745528 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.745525    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/298385fa-8136-472f-b285-962019f92ae9-istiod-ca-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.745769 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.745537    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/298385fa-8136-472f-b285-962019f92ae9-istio-data\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:18:58.872993 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.872961    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"]
Jul 23 16:18:58.876545 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:18:58.876519    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8f4ec811-gw-openshift-default-5b978d5999-swn4h"]
Jul 23 16:19:00.809970 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:00.809940    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="298385fa-8136-472f-b285-962019f92ae9" path="/var/lib/kubelet/pods/298385fa-8136-472f-b285-962019f92ae9/volumes"
Jul 23 16:19:04.222104 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:04.222072    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:19:12.208172 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:12.207141    2555 patch_prober.go:28] interesting pod/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:36862->10.133.0.43:15021: read: connection reset by peer" start-of-body=
Jul 23 16:19:12.208172 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:12.207196    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:36862->10.133.0.43:15021: read: connection reset by peer"
Jul 23 16:19:12.623117 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:12.623019    2555 generic.go:356] "Generic (PLEG): container finished" podID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerID="b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e" exitCode=0
Jul 23 16:19:12.623117 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:12.623093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" event={"ID":"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81","Type":"ContainerDied","Data":"b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e"}
Jul 23 16:19:12.623488 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:12.623472    2555 scope.go:117] "RemoveContainer" containerID="b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e"
Jul 23 16:19:12.624397 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:12.624373    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:12.624454 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:12.624440    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:12.624490 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:12.624475    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:13.629648 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:13.629614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" event={"ID":"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81","Type":"ContainerStarted","Data":"06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8"}
Jul 23 16:19:14.377602 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:14.377558    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:19:14.377838 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:14.377694    2555 patch_prober.go:28] interesting pod/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 23 16:19:14.377838 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:14.377739    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 23 16:19:15.377611 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:15.377563    2555 patch_prober.go:28] interesting pod/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 23 16:19:15.378120 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:15.377630    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 23 16:19:16.377311 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:16.377273    2555 patch_prober.go:28] interesting pod/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 23 16:19:16.377605 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:16.377575    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 23 16:19:17.376812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:17.376772    2555 patch_prober.go:28] interesting pod/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 23 16:19:17.377378 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:17.377344    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 23 16:19:18.380291 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:18.380262    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:19:18.380776 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:18.380644    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:19:18.381345 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:18.381327    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:19:36.422621 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:36.422584    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:19:39.310177 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:39.310140    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"]
Jul 23 16:19:39.310740 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:39.310394    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy" containerID="cri-o://06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8" gracePeriod=30
Jul 23 16:19:40.177397 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177365    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"]
Jul 23 16:19:40.177798 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177782    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy"
Jul 23 16:19:40.177857 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177800    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad861b4-1c59-43b9-a212-13fe15c2ff21" containerName="istio-proxy"
Jul 23 16:19:40.177857 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177813    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy"
Jul 23 16:19:40.177857 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177820    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy"
Jul 23 16:19:40.177857 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177830    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy"
Jul 23 16:19:40.177857 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177836    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy"
Jul 23 16:19:40.178012 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177890    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy"
Jul 23 16:19:40.178012 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.177900    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="298385fa-8136-472f-b285-962019f92ae9" containerName="istio-proxy"
Jul 23 16:19:40.180962 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.180946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.183635 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.183613    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-063a9ab3-openshift-default-dockercfg-n289m\""
Jul 23 16:19:40.195107 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.195080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"]
Jul 23 16:19:40.288355 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288311    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-certs\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.288577 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288366    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-envoy\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.288577 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-podinfo\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.288577 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288500    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-credential-socket\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.288577 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288546    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-token\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.288821 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288598    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istiod-ca-cert\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.288821 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288639    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-data\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.288821 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288686    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvjp\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-kube-api-access-tjvjp\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.288821 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.288720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-socket\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390074 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-data\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390074 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvjp\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-kube-api-access-tjvjp\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390568 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390181    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-socket\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390568 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-certs\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390568 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-envoy\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390568 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390273    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-podinfo\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390568 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-credential-socket\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390568 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-token\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.390568 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istiod-ca-cert\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.391033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390580    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-data\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.391033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390674    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-socket\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.391033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-credential-socket\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.391033 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.390969    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-certs\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.391188 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.391155    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istiod-ca-cert\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.392561 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.392539    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-envoy\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.392932 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.392909    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-podinfo\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.399121 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.399093    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvjp\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-kube-api-access-tjvjp\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.399370 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.399348    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-token\") pod \"e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") " pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.491329 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.491244    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:40.636108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.636075    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"]
Jul 23 16:19:40.638113 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:19:40.638084    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a94cd1c_59a9_4cf7_9d0b_3d017df1c89a.slice/crio-37c024dd71bb62436cc31e4879f3304ca6d857b4d42b17dfa11eb36f81f7601b WatchSource:0}: Error finding container 37c024dd71bb62436cc31e4879f3304ca6d857b4d42b17dfa11eb36f81f7601b: Status 404 returned error can't find the container with id 37c024dd71bb62436cc31e4879f3304ca6d857b4d42b17dfa11eb36f81f7601b
Jul 23 16:19:40.640357 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.640320    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:40.640451 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.640404    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:40.640451 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.640445    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:40.760395 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.760306    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" event={"ID":"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a","Type":"ContainerStarted","Data":"e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef"}
Jul 23 16:19:40.760395 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.760365    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" event={"ID":"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a","Type":"ContainerStarted","Data":"37c024dd71bb62436cc31e4879f3304ca6d857b4d42b17dfa11eb36f81f7601b"}
Jul 23 16:19:40.780097 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:40.780043    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" podStartSLOduration=0.78002468 podStartE2EDuration="780.02468ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:19:40.778970613 +0000 UTC m=+1868.404505089" watchObservedRunningTime="2026-07-23 16:19:40.78002468 +0000 UTC m=+1868.405559155"
Jul 23 16:19:41.491630 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:41.491590    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:41.496812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:41.496782    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:41.765500 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:41.765417    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:41.766324 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:41.766306    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:44.563016 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.562989    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:19:44.734442 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734404    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-credential-socket\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.734653 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734461    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-certs\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.734653 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734487    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-envoy\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.734653 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734509    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-socket\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.734653 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734545    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-token\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.734653 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734602    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istiod-ca-cert\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.734653 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734625    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzt6r\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-kube-api-access-tzt6r\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.735009 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734727    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:19:44.735009 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734821    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:19:44.735009 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734832    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-podinfo\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.735009 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734901    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-data\") pod \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\" (UID: \"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81\") "
Jul 23 16:19:44.735009 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.734845    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:19:44.735271 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.735032    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:19:44.735271 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.735257    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-credential-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:44.735377 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.735276    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:44.735377 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.735285    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-workload-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:44.735377 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.735293    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istiod-ca-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:44.735776 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.735733    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:19:44.736970 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.736945    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 16:19:44.737058 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.736974    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-kube-api-access-tzt6r" (OuterVolumeSpecName: "kube-api-access-tzt6r") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "kube-api-access-tzt6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:19:44.737217 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.737197    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:19:44.737276 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.737212    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" (UID: "9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:19:44.781848 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.781815    2555 generic.go:356] "Generic (PLEG): container finished" podID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerID="06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8" exitCode=0
Jul 23 16:19:44.782017 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.781868    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"
Jul 23 16:19:44.782017 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.781886    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" event={"ID":"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81","Type":"ContainerDied","Data":"06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8"}
Jul 23 16:19:44.782017 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.781960    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj" event={"ID":"9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81","Type":"ContainerDied","Data":"8fcf6725e57cf6baa83692a063091af21cec2a266e8683b8edb4400f5fa81b55"}
Jul 23 16:19:44.782017 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.781983    2555 scope.go:117] "RemoveContainer" containerID="06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8"
Jul 23 16:19:44.791053 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.791029    2555 scope.go:117] "RemoveContainer" containerID="b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e"
Jul 23 16:19:44.799128 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.799108    2555 scope.go:117] "RemoveContainer" containerID="06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8"
Jul 23 16:19:44.799386 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:19:44.799368    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8\": container with ID starting with 06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8 not found: ID does not exist" containerID="06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8"
Jul 23 16:19:44.799462 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.799398    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8"} err="failed to get container status \"06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8\": rpc error: code = NotFound desc = could not find container \"06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8\": container with ID starting with 06ccc8c744d64822ee25c866acf741b6d661bb0820d96ac83f391f897227c0b8 not found: ID does not exist"
Jul 23 16:19:44.799462 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.799426    2555 scope.go:117] "RemoveContainer" containerID="b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e"
Jul 23 16:19:44.799670 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:19:44.799655    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e\": container with ID starting with b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e not found: ID does not exist" containerID="b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e"
Jul 23 16:19:44.799709 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.799676    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e"} err="failed to get container status \"b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e\": rpc error: code = NotFound desc = could not find container \"b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e\": container with ID starting with b716075ca10f0215d863795e90ba2d67cad814a81db4b552a9d5335ff1af690e not found: ID does not exist"
Jul 23 16:19:44.803534 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.803512    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"]
Jul 23 16:19:44.809807 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.809788    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e475fc66-openshift-default-65fb4bcdb7-xmfvj"]
Jul 23 16:19:44.836585 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.836554    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:44.836585 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.836588    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzt6r\" (UniqueName: \"kubernetes.io/projected/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-kube-api-access-tzt6r\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:44.836772 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.836601    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-podinfo\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:44.836772 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.836610    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-data\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:44.836772 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:44.836619    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81-istio-envoy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:19:46.809665 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:46.809627    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" path="/var/lib/kubelet/pods/9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81/volumes"
Jul 23 16:19:48.919331 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:48.919297    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:19:57.346117 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:57.345879    2555 patch_prober.go:28] interesting pod/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": read tcp 10.133.0.2:41844->10.133.0.44:15021: read: connection reset by peer" start-of-body=
Jul 23 16:19:57.346117 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:57.345963    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": read tcp 10.133.0.2:41844->10.133.0.44:15021: read: connection reset by peer"
Jul 23 16:19:57.844110 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:57.844077    2555 generic.go:356] "Generic (PLEG): container finished" podID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerID="e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef" exitCode=0
Jul 23 16:19:57.844281 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:57.844149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" event={"ID":"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a","Type":"ContainerDied","Data":"e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef"}
Jul 23 16:19:57.844556 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:57.844543    2555 scope.go:117] "RemoveContainer" containerID="e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef"
Jul 23 16:19:57.845595 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:57.845566    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:57.845669 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:57.845651    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:57.845706 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:57.845696    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:19:58.850847 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:58.850813    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" event={"ID":"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a","Type":"ContainerStarted","Data":"3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc"}
Jul 23 16:19:59.491606 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:59.491562    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:19:59.491881 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:59.491853    2555 patch_prober.go:28] interesting pod/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 23 16:19:59.491943 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:19:59.491924    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 23 16:20:00.492415 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:00.492378    2555 patch_prober.go:28] interesting pod/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 23 16:20:00.492837 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:00.492443    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 23 16:20:01.492477 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:01.492435    2555 patch_prober.go:28] interesting pod/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 23 16:20:01.492904 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:01.492500    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 23 16:20:02.491885 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:02.491845    2555 patch_prober.go:28] interesting pod/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 23 16:20:02.492086 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:02.491922    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 23 16:20:03.495476 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:03.495441    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:20:03.495977 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:03.495867    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:20:03.496632 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:03.496614    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:20:20.520096 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:20.520061    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:20:22.336677 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:22.336638    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"]
Jul 23 16:20:22.337076 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:22.336882    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy" containerID="cri-o://3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc" gracePeriod=30
Jul 23 16:20:27.592100 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.592069    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:20:27.723631 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723599    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjvjp\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-kube-api-access-tjvjp\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.723853 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723640    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-socket\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.723853 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723670    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-credential-socket\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.723853 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723799    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istiod-ca-cert\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.724037 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723855    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-podinfo\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.724037 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723911    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-certs\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.724037 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723940    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-token\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.724037 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723975    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-envoy\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.724037 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.723998    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-data\") pod \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\" (UID: \"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a\") "
Jul 23 16:20:27.724037 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.724030    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:20:27.724342 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.724019    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:20:27.724342 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.724138    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:20:27.724342 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.724255    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:20:27.724465 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.724386    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:27.724465 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.724406    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-workload-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:27.724465 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.724421    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-credential-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:27.724934 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.724915    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:20:27.726332 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.726311    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:20:27.726428 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.726338    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 16:20:27.726428 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.726365    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:20:27.726428 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.726367    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-kube-api-access-tjvjp" (OuterVolumeSpecName: "kube-api-access-tjvjp") pod "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" (UID: "1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a"). InnerVolumeSpecName "kube-api-access-tjvjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:20:27.825301 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.825265    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-envoy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:27.825301 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.825291    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-data\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:27.825301 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.825301    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvjp\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-kube-api-access-tjvjp\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:27.825301 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.825310    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istiod-ca-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:27.825561 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.825319    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-podinfo\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:27.825561 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:27.825327    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a-istio-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:20:28.002535 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.002454    2555 generic.go:356] "Generic (PLEG): container finished" podID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerID="3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc" exitCode=0
Jul 23 16:20:28.002535 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.002526    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"
Jul 23 16:20:28.002732 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.002548    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" event={"ID":"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a","Type":"ContainerDied","Data":"3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc"}
Jul 23 16:20:28.002732 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.002625    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj" event={"ID":"1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a","Type":"ContainerDied","Data":"37c024dd71bb62436cc31e4879f3304ca6d857b4d42b17dfa11eb36f81f7601b"}
Jul 23 16:20:28.002732 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.002650    2555 scope.go:117] "RemoveContainer" containerID="3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc"
Jul 23 16:20:28.011431 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.011411    2555 scope.go:117] "RemoveContainer" containerID="e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef"
Jul 23 16:20:28.018922 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.018906    2555 scope.go:117] "RemoveContainer" containerID="3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc"
Jul 23 16:20:28.019209 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:20:28.019184    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc\": container with ID starting with 3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc not found: ID does not exist" containerID="3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc"
Jul 23 16:20:28.019311 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.019213    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc"} err="failed to get container status \"3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc\": rpc error: code = NotFound desc = could not find container \"3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc\": container with ID starting with 3c54da31710dbeeef6130649a277dd53e2818f21adf9f67566821fcef4ec36dc not found: ID does not exist"
Jul 23 16:20:28.019311 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.019230    2555 scope.go:117] "RemoveContainer" containerID="e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef"
Jul 23 16:20:28.019463 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:20:28.019446    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef\": container with ID starting with e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef not found: ID does not exist" containerID="e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef"
Jul 23 16:20:28.019529 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.019466    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef"} err="failed to get container status \"e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef\": rpc error: code = NotFound desc = could not find container \"e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef\": container with ID starting with e7a5f9be763eb4b06796a58f5b5b81ff7bd99709062b6c67755658bdf187dcef not found: ID does not exist"
Jul 23 16:20:28.029233 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.029211    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"]
Jul 23 16:20:28.033241 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.033222    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-063a9ab3-openshift-default-7559fd4767-pglfj"]
Jul 23 16:20:28.809735 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:20:28.809701    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" path="/var/lib/kubelet/pods/1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a/volumes"
Jul 23 16:21:49.117696 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:21:49.117606    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:21:56.310665 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:21:56.310620    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:22:05.769140 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769093    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"]
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769455    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769466    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769479    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769485    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769498    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769504    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769522    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769528    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769584    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769591    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a94cd1c-59a9-4cf7-9d0b-3d017df1c89a" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769598    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy"
Jul 23 16:22:05.769613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.769605    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4209e0-f0b3-4da9-8cfa-1b12e78b5f81" containerName="istio-proxy"
Jul 23 16:22:05.772548 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.772530    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.775259 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.775241    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b5c55cc-openshift-default-dockercfg-9pjcb\""
Jul 23 16:22:05.799151 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.799120    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"]
Jul 23 16:22:05.800716 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.800686    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-data\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.800917 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.800821    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istiod-ca-cert\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.800917 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.800883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-credential-socket\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.800917 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.800912    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-podinfo\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.801129 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.800965    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-token\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.801129 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.800998    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx8c\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-kube-api-access-ktx8c\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.801129 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.801037    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-envoy\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.801129 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.801066    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-certs\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.801129 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.801123    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-socket\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902196 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902151    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-envoy\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902196 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902201    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-certs\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-socket\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902270    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-data\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902329    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istiod-ca-cert\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-credential-socket\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-podinfo\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902418    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-token\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902452    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx8c\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-kube-api-access-ktx8c\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902679    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-certs\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902821    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-credential-socket\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.902969 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.902859    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-data\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.903044 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.903026    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-socket\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.903574 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.903545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istiod-ca-cert\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.905391 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.905367    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-envoy\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.905717 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.905694    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-podinfo\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.911823 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.911795    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx8c\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-kube-api-access-ktx8c\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:05.912165 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:05.912141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-token\") pod \"e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") " pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:06.082437 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.082344    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:06.219771 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.219727    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"]
Jul 23 16:22:06.221575 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:22:06.221542    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70e3a5c_e32e_48a1_a2c7_13a2d3ac17df.slice/crio-14fb8b66623c06705628ef630d60acd4bc54218ba2a9d984f2923ffa6fca57f2 WatchSource:0}: Error finding container 14fb8b66623c06705628ef630d60acd4bc54218ba2a9d984f2923ffa6fca57f2: Status 404 returned error can't find the container with id 14fb8b66623c06705628ef630d60acd4bc54218ba2a9d984f2923ffa6fca57f2
Jul 23 16:22:06.223868 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.223838    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:22:06.223955 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.223902    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:22:06.223955 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.223933    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:22:06.440074 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.440039    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" event={"ID":"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df","Type":"ContainerStarted","Data":"9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7"}
Jul 23 16:22:06.440266 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.440102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" event={"ID":"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df","Type":"ContainerStarted","Data":"14fb8b66623c06705628ef630d60acd4bc54218ba2a9d984f2923ffa6fca57f2"}
Jul 23 16:22:06.463415 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.463350    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" podStartSLOduration=1.463331502 podStartE2EDuration="1.463331502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:22:06.461845261 +0000 UTC m=+2014.087379739" watchObservedRunningTime="2026-07-23 16:22:06.463331502 +0000 UTC m=+2014.088865977"
Jul 23 16:22:06.918379 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:06.918339    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:22:07.083119 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:07.083075    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:07.089765 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:07.089722    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:07.445241 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:07.445207    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:07.446399 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:07.446379    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:18.112733 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:18.112697    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:22:22.446417 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:22.446377    2555 patch_prober.go:28] interesting pod/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 23 16:22:22.446890 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:22.446451    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 23 16:22:22.518600 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:22.518566    2555 generic.go:356] "Generic (PLEG): container finished" podID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerID="9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7" exitCode=0
Jul 23 16:22:22.518792 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:22.518643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" event={"ID":"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df","Type":"ContainerDied","Data":"9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7"}
Jul 23 16:22:22.519154 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:22.519134    2555 scope.go:117] "RemoveContainer" containerID="9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7"
Jul 23 16:22:22.520119 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:22.520094    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:22:22.520173 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:22.520158    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:22:22.520208 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:22.520185    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:22:23.525392 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:23.525358    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" event={"ID":"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df","Type":"ContainerStarted","Data":"f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25"}
Jul 23 16:22:24.083407 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:24.083372    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:24.083664 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:24.083640    2555 patch_prober.go:28] interesting pod/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 23 16:22:24.083722 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:24.083703    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 23 16:22:25.083317 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:25.083277    2555 patch_prober.go:28] interesting pod/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 23 16:22:25.083695 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:25.083332    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 23 16:22:26.082990 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:26.082950    2555 patch_prober.go:28] interesting pod/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 23 16:22:26.083184 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:26.083012    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 23 16:22:27.082870 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:27.082829    2555 patch_prober.go:28] interesting pod/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 23 16:22:27.083277 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:27.082890    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 23 16:22:28.086716 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:28.086686    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:28.087226 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:28.087076    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:28.087784 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:28.087766    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:34.308480 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:34.308443    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:22:42.327892 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:42.327860    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"]
Jul 23 16:22:42.328360 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:42.328146    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy" containerID="cri-o://f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25" gracePeriod=30
Jul 23 16:22:43.089734 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:43.089700    2555 patch_prober.go:28] interesting pod/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 16:22:43.089929 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:43.089783    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 16:22:47.575468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.575444    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:47.632039 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.632018    2555 generic.go:356] "Generic (PLEG): container finished" podID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerID="f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25" exitCode=0
Jul 23 16:22:47.632178 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.632068    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"
Jul 23 16:22:47.632178 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.632072    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" event={"ID":"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df","Type":"ContainerDied","Data":"f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25"}
Jul 23 16:22:47.632256 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.632218    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw" event={"ID":"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df","Type":"ContainerDied","Data":"14fb8b66623c06705628ef630d60acd4bc54218ba2a9d984f2923ffa6fca57f2"}
Jul 23 16:22:47.632256 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.632249    2555 scope.go:117] "RemoveContainer" containerID="f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25"
Jul 23 16:22:47.640566 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.640550    2555 scope.go:117] "RemoveContainer" containerID="9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7"
Jul 23 16:22:47.647698 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.647681    2555 scope.go:117] "RemoveContainer" containerID="f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25"
Jul 23 16:22:47.647974 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:22:47.647954    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25\": container with ID starting with f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25 not found: ID does not exist" containerID="f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25"
Jul 23 16:22:47.648022 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.647981    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25"} err="failed to get container status \"f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25\": rpc error: code = NotFound desc = could not find container \"f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25\": container with ID starting with f5f6c5adda0a979037528733c1114cbf710aa96c0a80d7243dd5faf8e2553f25 not found: ID does not exist"
Jul 23 16:22:47.648022 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.647998    2555 scope.go:117] "RemoveContainer" containerID="9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7"
Jul 23 16:22:47.648232 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:22:47.648215    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7\": container with ID starting with 9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7 not found: ID does not exist" containerID="9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7"
Jul 23 16:22:47.648285 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.648241    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7"} err="failed to get container status \"9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7\": rpc error: code = NotFound desc = could not find container \"9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7\": container with ID starting with 9c304994f46848407ce3586d02d3c848cd5264f8e0fde79e04b5a6a2c2049ae7 not found: ID does not exist"
Jul 23 16:22:47.679765 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.679708    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-certs\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.679877 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.679822    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-podinfo\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.679877 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.679861    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktx8c\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-kube-api-access-ktx8c\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.679991 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.679889    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-data\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.679991 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.679914    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-socket\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.679991 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.679937    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:22:47.679991 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.679953    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-token\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.679991 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.679984    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istiod-ca-cert\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.680240 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680085    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-credential-socket\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.680240 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680205    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:22:47.680345 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680322    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:22:47.680345 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680326    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-envoy\") pod \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\" (UID: \"c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df\") "
Jul 23 16:22:47.680446 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680390    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:22:47.680652 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680630    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-credential-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.680652 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680654    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.680652 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680669    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-workload-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.680652 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680682    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istiod-ca-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.680983 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.680880    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-data" (OuterVolumeSpecName: "istio-data") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:22:47.682068 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.682039    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-kube-api-access-ktx8c" (OuterVolumeSpecName: "kube-api-access-ktx8c") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "kube-api-access-ktx8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:22:47.682152 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.682135    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:22:47.682375 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.682348    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 16:22:47.682375 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.682356    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-token" (OuterVolumeSpecName: "istio-token") pod "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" (UID: "c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:22:47.781308 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.781259    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktx8c\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-kube-api-access-ktx8c\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.781308 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.781302    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-data\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.781308 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.781312    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.781308 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.781321    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-envoy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.781308 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.781330    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df-istio-podinfo\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:22:47.953660 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.953626    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"]
Jul 23 16:22:47.959420 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:47.959393    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b5c55cc-openshift-default-65dc459f9f-vtkfw"]
Jul 23 16:22:48.809365 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:22:48.809330    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" path="/var/lib/kubelet/pods/c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df/volumes"
Jul 23 16:23:02.293374 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.293292    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"]
Jul 23 16:23:02.293792 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.293612    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy"
Jul 23 16:23:02.293792 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.293623    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy"
Jul 23 16:23:02.293792 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.293632    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy"
Jul 23 16:23:02.293792 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.293638    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy"
Jul 23 16:23:02.293792 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.293695    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy"
Jul 23 16:23:02.293792 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.293706    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70e3a5c-e32e-48a1-a2c7-13a2d3ac17df" containerName="istio-proxy"
Jul 23 16:23:02.298005 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.297983    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.300397 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.300371    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4b72f23f-openshift-default-dockercfg-5xwzt\""
Jul 23 16:23:02.311006 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.310968    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"]
Jul 23 16:23:02.407814 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.407738    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-token\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.408036 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.407987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-socket\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.408106 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.408047    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-certs\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.408106 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.408070    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-data\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.408197 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.408109    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-podinfo\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.408245 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.408195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-credential-socket\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.408280 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.408243    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-envoy\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.408320 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.408283    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-kube-api-access-xnprh\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.408320 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.408310    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istiod-ca-cert\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509378 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509338    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-certs\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509378 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-data\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509656 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-podinfo\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509656 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509450    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-credential-socket\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509656 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509471    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-envoy\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509656 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509494    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-kube-api-access-xnprh\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509656 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509522    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istiod-ca-cert\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509656 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509573    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-token\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509656 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-socket\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509990 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509934    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-credential-socket\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.509990 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.509960    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-data\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.510232 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.510197    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-certs\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.510368 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.510311    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istiod-ca-cert\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.510368 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.510321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-socket\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.511958 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.511938    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-envoy\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.512173 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.512157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-podinfo\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.517322 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.517295    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-kube-api-access-xnprh\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.517528 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.517511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-token\") pod \"e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") " pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.608435 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.608343    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:02.750611 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.750569    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"]
Jul 23 16:23:02.753661 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:23:02.753622    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfef8478_9f65_4fe0_9fe8_6590f35664f7.slice/crio-35d1fc3c3ee6b2ad15fb82f4475f296dc9273293e31591e51acd2deef741eca9 WatchSource:0}: Error finding container 35d1fc3c3ee6b2ad15fb82f4475f296dc9273293e31591e51acd2deef741eca9: Status 404 returned error can't find the container with id 35d1fc3c3ee6b2ad15fb82f4475f296dc9273293e31591e51acd2deef741eca9
Jul 23 16:23:02.756322 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.756282    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:23:02.756426 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.756375    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:23:02.756486 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:02.756421    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:23:03.704899 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:03.704858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" event={"ID":"bfef8478-9f65-4fe0-9fe8-6590f35664f7","Type":"ContainerStarted","Data":"2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3"}
Jul 23 16:23:03.705445 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:03.704947    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" event={"ID":"bfef8478-9f65-4fe0-9fe8-6590f35664f7","Type":"ContainerStarted","Data":"35d1fc3c3ee6b2ad15fb82f4475f296dc9273293e31591e51acd2deef741eca9"}
Jul 23 16:23:03.726073 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:03.726017    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" podStartSLOduration=1.7259999050000001 podStartE2EDuration="1.725999905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:23:03.724416747 +0000 UTC m=+2071.349951223" watchObservedRunningTime="2026-07-23 16:23:03.725999905 +0000 UTC m=+2071.351534381"
Jul 23 16:23:04.609405 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:04.609366    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:04.614356 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:04.614330    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:04.709903 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:04.709868    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:04.710865 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:04.710848    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:12.731374 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:12.731337    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:23:20.751932 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:20.751887    2555 patch_prober.go:28] interesting pod/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 23 16:23:20.752316 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:20.751971    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 23 16:23:21.816011 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:21.815978    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerID="2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3" exitCode=0
Jul 23 16:23:21.816560 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:21.816035    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" event={"ID":"bfef8478-9f65-4fe0-9fe8-6590f35664f7","Type":"ContainerDied","Data":"2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3"}
Jul 23 16:23:21.816560 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:21.816517    2555 scope.go:117] "RemoveContainer" containerID="2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3"
Jul 23 16:23:21.817598 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:21.817565    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:23:21.817694 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:21.817635    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:23:21.817694 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:21.817664    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:23:22.822190 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:22.822150    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" event={"ID":"bfef8478-9f65-4fe0-9fe8-6590f35664f7","Type":"ContainerStarted","Data":"7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434"}
Jul 23 16:23:23.608807 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:23.608769    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:23.609014 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:23.608977    2555 patch_prober.go:28] interesting pod/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 23 16:23:23.609067 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:23.609021    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 23 16:23:24.609604 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:24.609568    2555 patch_prober.go:28] interesting pod/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 23 16:23:24.610032 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:24.609627    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 23 16:23:25.609178 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:25.609139    2555 patch_prober.go:28] interesting pod/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 23 16:23:25.609377 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:25.609197    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 23 16:23:25.720944 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:25.720908    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:23:26.609032 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:26.608992    2555 patch_prober.go:28] interesting pod/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 23 16:23:26.609221 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:26.609055    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 23 16:23:27.612682 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:27.612649    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:27.613077 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:27.612998    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:27.613729 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:27.613714    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:23:32.852064 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:32.852037    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:23:32.857055 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:32.857035    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:23:42.522173 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:23:42.522136    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:24:03.417339 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:03.417303    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:24:13.815825 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:13.815787    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:24:44.001741 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:44.001658    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"]
Jul 23 16:24:44.177902 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:44.177863    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy" containerID="cri-o://7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434" gracePeriod=30
Jul 23 16:24:49.441449 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.441374    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:24:49.586451 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586417    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-data\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.586451 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586457    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istiod-ca-cert\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.586723 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586478    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-credential-socket\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.586723 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586507    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-token\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.586723 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586555    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-certs\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.586723 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586590    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-socket\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.586723 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586618    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-podinfo\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.586723 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586634    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-envoy\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.586723 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586654    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-kube-api-access-xnprh\") pod \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\" (UID: \"bfef8478-9f65-4fe0-9fe8-6590f35664f7\") "
Jul 23 16:24:49.587119 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586782    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:24:49.587119 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586904    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:24:49.587119 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.586991    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istiod-ca-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:49.587119 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.587010    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-credential-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:49.587119 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.587059    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:24:49.587348 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.587302    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:24:49.587444 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.587421    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:24:49.589118 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.589088    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-kube-api-access-xnprh" (OuterVolumeSpecName: "kube-api-access-xnprh") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "kube-api-access-xnprh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:24:49.589310 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.589290    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:24:49.589591 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.589575    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:24:49.589931 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.589909    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bfef8478-9f65-4fe0-9fe8-6590f35664f7" (UID: "bfef8478-9f65-4fe0-9fe8-6590f35664f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 16:24:49.688227 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.688187    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-data\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:49.688227 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.688226    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:49.688434 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.688242    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:49.688434 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.688254    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-workload-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:49.688434 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.688267    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-podinfo\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:49.688434 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.688279    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfef8478-9f65-4fe0-9fe8-6590f35664f7-istio-envoy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:49.688434 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:49.688292    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnprh\" (UniqueName: \"kubernetes.io/projected/bfef8478-9f65-4fe0-9fe8-6590f35664f7-kube-api-access-xnprh\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:24:50.206687 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.206647    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerID="7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434" exitCode=0
Jul 23 16:24:50.206901 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.206713    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"
Jul 23 16:24:50.206901 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.206740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" event={"ID":"bfef8478-9f65-4fe0-9fe8-6590f35664f7","Type":"ContainerDied","Data":"7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434"}
Jul 23 16:24:50.206901 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.206829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v" event={"ID":"bfef8478-9f65-4fe0-9fe8-6590f35664f7","Type":"ContainerDied","Data":"35d1fc3c3ee6b2ad15fb82f4475f296dc9273293e31591e51acd2deef741eca9"}
Jul 23 16:24:50.206901 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.206852    2555 scope.go:117] "RemoveContainer" containerID="7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434"
Jul 23 16:24:50.215962 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.215943    2555 scope.go:117] "RemoveContainer" containerID="2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3"
Jul 23 16:24:50.223765 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.223728    2555 scope.go:117] "RemoveContainer" containerID="7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434"
Jul 23 16:24:50.224044 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:24:50.224025    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434\": container with ID starting with 7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434 not found: ID does not exist" containerID="7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434"
Jul 23 16:24:50.224122 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.224051    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434"} err="failed to get container status \"7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434\": rpc error: code = NotFound desc = could not find container \"7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434\": container with ID starting with 7ba51e19ebe64b6b359b2df4e0e75e9db90d7a21ad69331bbc385b30a645e434 not found: ID does not exist"
Jul 23 16:24:50.224122 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.224082    2555 scope.go:117] "RemoveContainer" containerID="2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3"
Jul 23 16:24:50.224333 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:24:50.224314    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3\": container with ID starting with 2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3 not found: ID does not exist" containerID="2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3"
Jul 23 16:24:50.224384 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.224339    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3"} err="failed to get container status \"2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3\": rpc error: code = NotFound desc = could not find container \"2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3\": container with ID starting with 2aae83b1b0342a28443f474951b899cce721629dc075613984a90ca55227f0b3 not found: ID does not exist"
Jul 23 16:24:50.227443 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.227423    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"]
Jul 23 16:24:50.230959 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.230938    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4b72f23f-openshift-default-77bb9bccf9-k8x7v"]
Jul 23 16:24:50.810683 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:50.810650    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" path="/var/lib/kubelet/pods/bfef8478-9f65-4fe0-9fe8-6590f35664f7/volumes"
Jul 23 16:24:54.617889 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:24:54.617859    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:25:02.611594 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:02.611557    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:25:40.303007 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.302974    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"]
Jul 23 16:25:40.303418 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.303303    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy"
Jul 23 16:25:40.303418 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.303314    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy"
Jul 23 16:25:40.303418 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.303403    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy"
Jul 23 16:25:40.303418 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.303415    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy"
Jul 23 16:25:40.306409 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.306391    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.308792 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.308766    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e51bfc-openshift-default-dockercfg-p8f6f\""
Jul 23 16:25:40.317951 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.317913    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"]
Jul 23 16:25:40.447083 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447045    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-socket\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.447278 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447089    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-envoy\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.447278 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447137    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-certs\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.447278 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447165    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05feaa83-7cfc-4421-9873-e606742b375b-istiod-ca-cert\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.447278 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4w8\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-kube-api-access-km4w8\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.447278 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05feaa83-7cfc-4421-9873-e606742b375b-istio-podinfo\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.447278 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-data\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.447513 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447333    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-credential-socket\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.447513 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.447384    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-istio-token\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548427 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05feaa83-7cfc-4421-9873-e606742b375b-istio-podinfo\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548427 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-data\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548694 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548473    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-credential-socket\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548694 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548504    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-istio-token\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548694 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-socket\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548694 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548611    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-envoy\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548694 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548668    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-certs\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548996 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05feaa83-7cfc-4421-9873-e606742b375b-istiod-ca-cert\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548996 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4w8\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-kube-api-access-km4w8\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548996 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548892    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-data\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548996 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548935    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-credential-socket\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.548996 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.548956    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-socket\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.549254 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.549125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-certs\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.549502 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.549477    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05feaa83-7cfc-4421-9873-e606742b375b-istiod-ca-cert\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.551058 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.551040    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-envoy\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.551187 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.551167    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05feaa83-7cfc-4421-9873-e606742b375b-istio-podinfo\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.556734 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.556662    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-istio-token\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.556960 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.556942    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4w8\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-kube-api-access-km4w8\") pod \"e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") " pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.617650 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.617619    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:40.778457 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.778430    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"]
Jul 23 16:25:40.780585 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:25:40.780556    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05feaa83_7cfc_4421_9873_e606742b375b.slice/crio-fd3553eec73c06428e9c11b7cf828c6e1692ff155617fb76c99145bdcf0ed781 WatchSource:0}: Error finding container fd3553eec73c06428e9c11b7cf828c6e1692ff155617fb76c99145bdcf0ed781: Status 404 returned error can't find the container with id fd3553eec73c06428e9c11b7cf828c6e1692ff155617fb76c99145bdcf0ed781
Jul 23 16:25:40.782631 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.782599    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:25:40.782743 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.782677    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:25:40.782743 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:40.782720    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:25:41.436652 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:41.436618    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" event={"ID":"05feaa83-7cfc-4421-9873-e606742b375b","Type":"ContainerStarted","Data":"c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08"}
Jul 23 16:25:41.437057 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:41.436696    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" event={"ID":"05feaa83-7cfc-4421-9873-e606742b375b","Type":"ContainerStarted","Data":"fd3553eec73c06428e9c11b7cf828c6e1692ff155617fb76c99145bdcf0ed781"}
Jul 23 16:25:41.455931 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:41.455884    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" podStartSLOduration=1.455867446 podStartE2EDuration="1.455867446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:25:41.454545523 +0000 UTC m=+2229.080079997" watchObservedRunningTime="2026-07-23 16:25:41.455867446 +0000 UTC m=+2229.081401921"
Jul 23 16:25:41.618045 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:41.618004    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:41.623364 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:41.623335    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:42.441295 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:42.441262    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:42.442343 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:42.442322    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:48.214504 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:48.214466    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:25:56.510340 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:56.510304    2555 generic.go:356] "Generic (PLEG): container finished" podID="05feaa83-7cfc-4421-9873-e606742b375b" containerID="c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08" exitCode=0
Jul 23 16:25:56.510810 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:56.510379    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" event={"ID":"05feaa83-7cfc-4421-9873-e606742b375b","Type":"ContainerDied","Data":"c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08"}
Jul 23 16:25:56.510861 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:56.510840    2555 scope.go:117] "RemoveContainer" containerID="c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08"
Jul 23 16:25:56.511847 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:56.511814    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:25:56.511954 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:56.511880    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:25:56.511954 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:56.511913    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 23 16:25:57.441929 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:57.441886    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:57.516575 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:57.516539    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" event={"ID":"05feaa83-7cfc-4421-9873-e606742b375b","Type":"ContainerStarted","Data":"8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32"}
Jul 23 16:25:57.618703 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:57.618667    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:25:57.619808 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:57.619777    2555 patch_prober.go:28] interesting pod/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 23 16:25:57.619915 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:57.619852    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 23 16:25:58.548021 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:58.547992    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:25:58.618142 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:58.618099    2555 patch_prober.go:28] interesting pod/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 23 16:25:58.618325 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:58.618165    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 23 16:25:59.618010 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:59.617973    2555 patch_prober.go:28] interesting pod/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 23 16:25:59.618460 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:25:59.618043    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 23 16:26:00.618665 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:00.618631    2555 patch_prober.go:28] interesting pod/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 23 16:26:00.619077 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:00.618717    2555 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 23 16:26:01.622333 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:01.622299    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:26:01.622817 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:01.622576    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:26:01.623228 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:01.623212    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:26:17.816270 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:17.816233    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:26:35.317516 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.317481    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"]
Jul 23 16:26:35.317987 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.317826    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy"
Jul 23 16:26:35.317987 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.317838    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfef8478-9f65-4fe0-9fe8-6590f35664f7" containerName="istio-proxy"
Jul 23 16:26:35.321076 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.321057    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.324808 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.324788    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-799a66\"/\"default-dockercfg-m4b6r\""
Jul 23 16:26:35.324808 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.324801    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-799a66\"/\"test-model-799a66-kserve-self-signed-certs\""
Jul 23 16:26:35.324974 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.324809    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-799a66\"/\"openshift-service-ca.crt\""
Jul 23 16:26:35.324974 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.324853    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-799a66\"/\"kube-root-ca.crt\""
Jul 23 16:26:35.332531 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.332511    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"]
Jul 23 16:26:35.457156 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.457111    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-tmp-dir\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.457156 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.457156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-dshm\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.457432 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.457177    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-home\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.457432 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.457317    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637dd507-0a2d-45b0-af8f-5c81743b797b-tls-certs\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.457432 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.457361    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxg4d\" (UniqueName: \"kubernetes.io/projected/637dd507-0a2d-45b0-af8f-5c81743b797b-kube-api-access-fxg4d\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.457432 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.457402    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-model-cache\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.558260 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637dd507-0a2d-45b0-af8f-5c81743b797b-tls-certs\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.558260 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558263    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxg4d\" (UniqueName: \"kubernetes.io/projected/637dd507-0a2d-45b0-af8f-5c81743b797b-kube-api-access-fxg4d\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.558260 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558283    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-model-cache\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.558595 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-tmp-dir\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.558595 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-dshm\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.558595 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-home\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.558899 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-model-cache\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.558979 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558930    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-tmp-dir\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.559047 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.558974    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-home\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.560810 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.560788    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-dshm\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.561058 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.561042    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637dd507-0a2d-45b0-af8f-5c81743b797b-tls-certs\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.566652 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.566630    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxg4d\" (UniqueName: \"kubernetes.io/projected/637dd507-0a2d-45b0-af8f-5c81743b797b-kube-api-access-fxg4d\") pod \"test-model-799a66-kserve-79c464ff6f-rc2qb\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") " pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.630699 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.630594    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:35.762898 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:35.762873    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"]
Jul 23 16:26:35.764970 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:26:35.764941    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod637dd507_0a2d_45b0_af8f_5c81743b797b.slice/crio-5952b85777d77779d5f6d6b3199cfcfc8f0b82984519f8cd08eeec91e904a13c WatchSource:0}: Error finding container 5952b85777d77779d5f6d6b3199cfcfc8f0b82984519f8cd08eeec91e904a13c: Status 404 returned error can't find the container with id 5952b85777d77779d5f6d6b3199cfcfc8f0b82984519f8cd08eeec91e904a13c
Jul 23 16:26:36.695052 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:36.695016    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb" event={"ID":"637dd507-0a2d-45b0-af8f-5c81743b797b","Type":"ContainerStarted","Data":"5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154"}
Jul 23 16:26:36.695486 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:36.695061    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb" event={"ID":"637dd507-0a2d-45b0-af8f-5c81743b797b","Type":"ContainerStarted","Data":"5952b85777d77779d5f6d6b3199cfcfc8f0b82984519f8cd08eeec91e904a13c"}
Jul 23 16:26:36.738549 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:36.738497    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb" podStartSLOduration=1.738480008 podStartE2EDuration="1.738480008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:26:36.732507766 +0000 UTC m=+2284.358042242" watchObservedRunningTime="2026-07-23 16:26:36.738480008 +0000 UTC m=+2284.364014463"
Jul 23 16:26:40.927409 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:40.927372    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:26:45.631063 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:45.631005    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:45.631505 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:45.631087    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:45.643734 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:45.643707    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:26:45.747243 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:26:45.747213    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:27:21.073874 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:21.073843    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"]
Jul 23 16:27:27.007199 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:27.007113    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:27:36.094130 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.094079    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb" podUID="637dd507-0a2d-45b0-af8f-5c81743b797b" containerName="main" containerID="cri-o://5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154" gracePeriod=45
Jul 23 16:27:36.349834 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.349769    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:27:36.421309 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421274    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-home\") pod \"637dd507-0a2d-45b0-af8f-5c81743b797b\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") "
Jul 23 16:27:36.421492 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421363    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-model-cache\") pod \"637dd507-0a2d-45b0-af8f-5c81743b797b\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") "
Jul 23 16:27:36.421492 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421387    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-tmp-dir\") pod \"637dd507-0a2d-45b0-af8f-5c81743b797b\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") "
Jul 23 16:27:36.421492 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421407    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-dshm\") pod \"637dd507-0a2d-45b0-af8f-5c81743b797b\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") "
Jul 23 16:27:36.421492 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421425    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxg4d\" (UniqueName: \"kubernetes.io/projected/637dd507-0a2d-45b0-af8f-5c81743b797b-kube-api-access-fxg4d\") pod \"637dd507-0a2d-45b0-af8f-5c81743b797b\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") "
Jul 23 16:27:36.421492 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421457    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637dd507-0a2d-45b0-af8f-5c81743b797b-tls-certs\") pod \"637dd507-0a2d-45b0-af8f-5c81743b797b\" (UID: \"637dd507-0a2d-45b0-af8f-5c81743b797b\") "
Jul 23 16:27:36.421793 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421580    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-home" (OuterVolumeSpecName: "home") pod "637dd507-0a2d-45b0-af8f-5c81743b797b" (UID: "637dd507-0a2d-45b0-af8f-5c81743b797b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:36.421793 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421661    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-model-cache" (OuterVolumeSpecName: "model-cache") pod "637dd507-0a2d-45b0-af8f-5c81743b797b" (UID: "637dd507-0a2d-45b0-af8f-5c81743b797b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:36.421793 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421680    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "637dd507-0a2d-45b0-af8f-5c81743b797b" (UID: "637dd507-0a2d-45b0-af8f-5c81743b797b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:36.421929 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421799    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:36.421929 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421817    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:36.421929 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.421833    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:36.423702 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.423676    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/637dd507-0a2d-45b0-af8f-5c81743b797b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "637dd507-0a2d-45b0-af8f-5c81743b797b" (UID: "637dd507-0a2d-45b0-af8f-5c81743b797b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 16:27:36.423836 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.423818    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-dshm" (OuterVolumeSpecName: "dshm") pod "637dd507-0a2d-45b0-af8f-5c81743b797b" (UID: "637dd507-0a2d-45b0-af8f-5c81743b797b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:36.423901 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.423833    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637dd507-0a2d-45b0-af8f-5c81743b797b-kube-api-access-fxg4d" (OuterVolumeSpecName: "kube-api-access-fxg4d") pod "637dd507-0a2d-45b0-af8f-5c81743b797b" (UID: "637dd507-0a2d-45b0-af8f-5c81743b797b"). InnerVolumeSpecName "kube-api-access-fxg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:27:36.522283 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.522242    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637dd507-0a2d-45b0-af8f-5c81743b797b-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:36.522283 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.522275    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxg4d\" (UniqueName: \"kubernetes.io/projected/637dd507-0a2d-45b0-af8f-5c81743b797b-kube-api-access-fxg4d\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:36.522283 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.522285    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637dd507-0a2d-45b0-af8f-5c81743b797b-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:36.962308 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.962275    2555 generic.go:356] "Generic (PLEG): container finished" podID="637dd507-0a2d-45b0-af8f-5c81743b797b" containerID="5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154" exitCode=0
Jul 23 16:27:36.962497 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.962318    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"
Jul 23 16:27:36.962497 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.962361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb" event={"ID":"637dd507-0a2d-45b0-af8f-5c81743b797b","Type":"ContainerDied","Data":"5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154"}
Jul 23 16:27:36.962497 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.962405    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb" event={"ID":"637dd507-0a2d-45b0-af8f-5c81743b797b","Type":"ContainerDied","Data":"5952b85777d77779d5f6d6b3199cfcfc8f0b82984519f8cd08eeec91e904a13c"}
Jul 23 16:27:36.962497 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.962423    2555 scope.go:117] "RemoveContainer" containerID="5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154"
Jul 23 16:27:36.970714 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.970696    2555 scope.go:117] "RemoveContainer" containerID="5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154"
Jul 23 16:27:36.971000 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:27:36.970982    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154\": container with ID starting with 5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154 not found: ID does not exist" containerID="5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154"
Jul 23 16:27:36.971062 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.971008    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154"} err="failed to get container status \"5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154\": rpc error: code = NotFound desc = could not find container \"5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154\": container with ID starting with 5353df88ccef2def4e40d59923a6e460fe7e938b5ac8044024c4213b25130154 not found: ID does not exist"
Jul 23 16:27:36.981881 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.981850    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"]
Jul 23 16:27:36.987535 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:36.987509    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-799a66/test-model-799a66-kserve-79c464ff6f-rc2qb"]
Jul 23 16:27:37.118852 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:37.118741    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:27:38.810065 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:38.810026    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637dd507-0a2d-45b0-af8f-5c81743b797b" path="/var/lib/kubelet/pods/637dd507-0a2d-45b0-af8f-5c81743b797b/volumes"
Jul 23 16:27:43.237570 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:43.237533    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"]
Jul 23 16:27:43.237999 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:43.237744    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy" containerID="cri-o://8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32" gracePeriod=30
Jul 23 16:27:46.625344 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:46.625310    2555 patch_prober.go:28] interesting pod/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 16:27:46.625772 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:46.625380    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 16:27:48.496599 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.496574    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:27:48.629851 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.629744    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-credential-socket\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.629851 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.629812    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km4w8\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-kube-api-access-km4w8\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.629851 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.629837    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05feaa83-7cfc-4421-9873-e606742b375b-istio-podinfo\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.629851 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.629854    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-data\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.630201 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.629869    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-istio-token\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.630201 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.629894    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05feaa83-7cfc-4421-9873-e606742b375b-istiod-ca-cert\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.630201 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.629939    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-envoy\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.630201 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.629972    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-certs\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.630201 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.630032    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-socket\") pod \"05feaa83-7cfc-4421-9873-e606742b375b\" (UID: \"05feaa83-7cfc-4421-9873-e606742b375b\") "
Jul 23 16:27:48.630201 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.630067    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:48.630502 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.630285    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:48.630502 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.630302    2555 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-credential-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:48.630502 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.630473    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:48.630658 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.630592    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05feaa83-7cfc-4421-9873-e606742b375b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:27:48.630868 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.630845    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-data" (OuterVolumeSpecName: "istio-data") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:48.632735 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.632711    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05feaa83-7cfc-4421-9873-e606742b375b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 16:27:48.632834 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.632733    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-kube-api-access-km4w8" (OuterVolumeSpecName: "kube-api-access-km4w8") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "kube-api-access-km4w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:27:48.632834 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.632793    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:27:48.632834 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.632793    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-istio-token" (OuterVolumeSpecName: "istio-token") pod "05feaa83-7cfc-4421-9873-e606742b375b" (UID: "05feaa83-7cfc-4421-9873-e606742b375b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:27:48.731613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.731572    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4w8\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-kube-api-access-km4w8\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:48.731613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.731606    2555 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05feaa83-7cfc-4421-9873-e606742b375b-istio-podinfo\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:48.731613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.731617    2555 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-data\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:48.731613 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.731625    2555 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05feaa83-7cfc-4421-9873-e606742b375b-istio-token\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:48.731921 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.731633    2555 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05feaa83-7cfc-4421-9873-e606742b375b-istiod-ca-cert\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:48.731921 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.731641    2555 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-istio-envoy\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:48.731921 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.731648    2555 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:48.731921 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:48.731655    2555 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05feaa83-7cfc-4421-9873-e606742b375b-workload-socket\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:27:49.015366 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.015333    2555 generic.go:356] "Generic (PLEG): container finished" podID="05feaa83-7cfc-4421-9873-e606742b375b" containerID="8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32" exitCode=0
Jul 23 16:27:49.015578 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.015376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" event={"ID":"05feaa83-7cfc-4421-9873-e606742b375b","Type":"ContainerDied","Data":"8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32"}
Jul 23 16:27:49.015578 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.015416    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv" event={"ID":"05feaa83-7cfc-4421-9873-e606742b375b","Type":"ContainerDied","Data":"fd3553eec73c06428e9c11b7cf828c6e1692ff155617fb76c99145bdcf0ed781"}
Jul 23 16:27:49.015578 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.015439    2555 scope.go:117] "RemoveContainer" containerID="8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32"
Jul 23 16:27:49.015578 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.015378    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"
Jul 23 16:27:49.026459 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.026439    2555 scope.go:117] "RemoveContainer" containerID="c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08"
Jul 23 16:27:49.039732 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.036046    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"]
Jul 23 16:27:49.040581 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.040558    2555 scope.go:117] "RemoveContainer" containerID="8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32"
Jul 23 16:27:49.041268 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:27:49.041237    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32\": container with ID starting with 8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32 not found: ID does not exist" containerID="8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32"
Jul 23 16:27:49.041381 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.041278    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32"} err="failed to get container status \"8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32\": rpc error: code = NotFound desc = could not find container \"8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32\": container with ID starting with 8365fcc6a42d4d73218cd482ced13d2552f961bb5b9646d8ed5990127643cd32 not found: ID does not exist"
Jul 23 16:27:49.041381 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.041304    2555 scope.go:117] "RemoveContainer" containerID="c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08"
Jul 23 16:27:49.041665 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:27:49.041647    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08\": container with ID starting with c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08 not found: ID does not exist" containerID="c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08"
Jul 23 16:27:49.041736 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.041668    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08"} err="failed to get container status \"c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08\": rpc error: code = NotFound desc = could not find container \"c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08\": container with ID starting with c554bc438670572415d5660239a4fa566fe80624f5480d1253595e0c7a810d08 not found: ID does not exist"
Jul 23 16:27:49.043481 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:49.043469    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e51bfc-openshift-default-67f8864c8f-dclwv"]
Jul 23 16:27:50.809651 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:50.809615    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05feaa83-7cfc-4421-9873-e606742b375b" path="/var/lib/kubelet/pods/05feaa83-7cfc-4421-9873-e606742b375b/volumes"
Jul 23 16:27:57.117437 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:27:57.117403    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:28:15.913683 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:15.913643    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:28:29.438336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438302    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"]
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438616    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637dd507-0a2d-45b0-af8f-5c81743b797b" containerName="main"
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438627    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="637dd507-0a2d-45b0-af8f-5c81743b797b" containerName="main"
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438642    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy"
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438647    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy"
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438660    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy"
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438666    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy"
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438716    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="637dd507-0a2d-45b0-af8f-5c81743b797b" containerName="main"
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438724    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy"
Jul 23 16:28:29.438863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.438731    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05feaa83-7cfc-4421-9873-e606742b375b" containerName="istio-proxy"
Jul 23 16:28:29.441633 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.441618    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"
Jul 23 16:28:29.444011 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.443984    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6jwk\""
Jul 23 16:28:29.444139 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.444014    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 16:28:29.448235 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.448211    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"]
Jul 23 16:28:29.589046 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.589009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601457a3-965d-44ab-a5dc-26ba19662771-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb\" (UID: \"601457a3-965d-44ab-a5dc-26ba19662771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"
Jul 23 16:28:29.689768 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.689655    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601457a3-965d-44ab-a5dc-26ba19662771-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb\" (UID: \"601457a3-965d-44ab-a5dc-26ba19662771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"
Jul 23 16:28:29.690272 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.690253    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601457a3-965d-44ab-a5dc-26ba19662771-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb\" (UID: \"601457a3-965d-44ab-a5dc-26ba19662771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"
Jul 23 16:28:29.753693 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.753659    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"
Jul 23 16:28:29.890938 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.887087    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"]
Jul 23 16:28:29.890938 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:28:29.888805    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601457a3_965d_44ab_a5dc_26ba19662771.slice/crio-b4303e083148772da8fbe1e94247cf8bdc39ae838c9e1a5aaa2d29c0119beca2 WatchSource:0}: Error finding container b4303e083148772da8fbe1e94247cf8bdc39ae838c9e1a5aaa2d29c0119beca2: Status 404 returned error can't find the container with id b4303e083148772da8fbe1e94247cf8bdc39ae838c9e1a5aaa2d29c0119beca2
Jul 23 16:28:29.890938 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:29.890416    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 16:28:30.198479 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:30.198441    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb" event={"ID":"601457a3-965d-44ab-a5dc-26ba19662771","Type":"ContainerStarted","Data":"b4303e083148772da8fbe1e94247cf8bdc39ae838c9e1a5aaa2d29c0119beca2"}
Jul 23 16:28:32.015656 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:32.015621    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:28:32.880957 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:32.880930    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:28:32.886162 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:32.886135    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:28:33.214220 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:33.214180    2555 generic.go:356] "Generic (PLEG): container finished" podID="601457a3-965d-44ab-a5dc-26ba19662771" containerID="ab351eb764b4d8462c6195484288742ec5beedbf5df3cd3804204e7ba842ed9c" exitCode=0
Jul 23 16:28:33.214609 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:33.214263    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb" event={"ID":"601457a3-965d-44ab-a5dc-26ba19662771","Type":"ContainerDied","Data":"ab351eb764b4d8462c6195484288742ec5beedbf5df3cd3804204e7ba842ed9c"}
Jul 23 16:28:34.355637 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:34.355607    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"
Jul 23 16:28:34.436265 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:34.436229    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601457a3-965d-44ab-a5dc-26ba19662771-maas-api-service-ca\") pod \"601457a3-965d-44ab-a5dc-26ba19662771\" (UID: \"601457a3-965d-44ab-a5dc-26ba19662771\") "
Jul 23 16:28:34.436612 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:34.436586    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601457a3-965d-44ab-a5dc-26ba19662771-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "601457a3-965d-44ab-a5dc-26ba19662771" (UID: "601457a3-965d-44ab-a5dc-26ba19662771"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:28:34.537717 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:34.537644    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/601457a3-965d-44ab-a5dc-26ba19662771-maas-api-service-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:28:35.224640 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:35.224615    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"
Jul 23 16:28:35.224933 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:35.224603    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb" event={"ID":"601457a3-965d-44ab-a5dc-26ba19662771","Type":"ContainerDied","Data":"b4303e083148772da8fbe1e94247cf8bdc39ae838c9e1a5aaa2d29c0119beca2"}
Jul 23 16:28:35.224933 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:35.224745    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4303e083148772da8fbe1e94247cf8bdc39ae838c9e1a5aaa2d29c0119beca2"
Jul 23 16:28:48.513680 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:48.513643    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:28:49.515306 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:49.515272    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"]
Jul 23 16:28:49.518113 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:49.518084    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-89bfd8-vdlvb"]
Jul 23 16:28:50.809764 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:50.809723    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="601457a3-965d-44ab-a5dc-26ba19662771" path="/var/lib/kubelet/pods/601457a3-965d-44ab-a5dc-26ba19662771/volumes"
Jul 23 16:28:56.426091 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.426001    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"]
Jul 23 16:28:56.426460 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.426388    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601457a3-965d-44ab-a5dc-26ba19662771" containerName="revoke-keys"
Jul 23 16:28:56.426460 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.426402    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="601457a3-965d-44ab-a5dc-26ba19662771" containerName="revoke-keys"
Jul 23 16:28:56.426537 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.426467    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="601457a3-965d-44ab-a5dc-26ba19662771" containerName="revoke-keys"
Jul 23 16:28:56.430037 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.430020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"
Jul 23 16:28:56.432428 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.432401    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6jwk\""
Jul 23 16:28:56.432580 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.432453    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 16:28:56.437499 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.437333    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"]
Jul 23 16:28:56.509195 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.509163    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf1982-c332-413c-9a9e-ab1aa82d4d21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx\" (UID: \"3baf1982-c332-413c-9a9e-ab1aa82d4d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"
Jul 23 16:28:56.610091 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.610051    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf1982-c332-413c-9a9e-ab1aa82d4d21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx\" (UID: \"3baf1982-c332-413c-9a9e-ab1aa82d4d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"
Jul 23 16:28:56.610671 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.610648    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf1982-c332-413c-9a9e-ab1aa82d4d21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx\" (UID: \"3baf1982-c332-413c-9a9e-ab1aa82d4d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"
Jul 23 16:28:56.741430 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.741332    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"
Jul 23 16:28:56.872487 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:56.872455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"]
Jul 23 16:28:56.875888 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:28:56.875859    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3baf1982_c332_413c_9a9e_ab1aa82d4d21.slice/crio-9962a07de3a1e5c135f519373a2156a001d0cca272fff52e9737037c9881f025 WatchSource:0}: Error finding container 9962a07de3a1e5c135f519373a2156a001d0cca272fff52e9737037c9881f025: Status 404 returned error can't find the container with id 9962a07de3a1e5c135f519373a2156a001d0cca272fff52e9737037c9881f025
Jul 23 16:28:57.323922 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:57.323820    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx" event={"ID":"3baf1982-c332-413c-9a9e-ab1aa82d4d21","Type":"ContainerStarted","Data":"5c4024b0c9e2e1a547ff968824a469277756421980b743b464a9e0d32a998320"}
Jul 23 16:28:57.323922 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:57.323884    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx" event={"ID":"3baf1982-c332-413c-9a9e-ab1aa82d4d21","Type":"ContainerStarted","Data":"9962a07de3a1e5c135f519373a2156a001d0cca272fff52e9737037c9881f025"}
Jul 23 16:28:57.340594 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:57.340549    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx" podStartSLOduration=1.340532992 podStartE2EDuration="1.340532992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:28:57.338967386 +0000 UTC m=+2424.964501863" watchObservedRunningTime="2026-07-23 16:28:57.340532992 +0000 UTC m=+2424.966067468"
Jul 23 16:28:58.330044 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:58.330008    2555 generic.go:356] "Generic (PLEG): container finished" podID="3baf1982-c332-413c-9a9e-ab1aa82d4d21" containerID="5c4024b0c9e2e1a547ff968824a469277756421980b743b464a9e0d32a998320" exitCode=0
Jul 23 16:28:58.330433 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:58.330090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx" event={"ID":"3baf1982-c332-413c-9a9e-ab1aa82d4d21","Type":"ContainerDied","Data":"5c4024b0c9e2e1a547ff968824a469277756421980b743b464a9e0d32a998320"}
Jul 23 16:28:59.467705 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:59.467677    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"
Jul 23 16:28:59.537172 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:59.537139    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf1982-c332-413c-9a9e-ab1aa82d4d21-maas-api-service-ca\") pod \"3baf1982-c332-413c-9a9e-ab1aa82d4d21\" (UID: \"3baf1982-c332-413c-9a9e-ab1aa82d4d21\") "
Jul 23 16:28:59.537500 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:59.537475    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3baf1982-c332-413c-9a9e-ab1aa82d4d21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3baf1982-c332-413c-9a9e-ab1aa82d4d21" (UID: "3baf1982-c332-413c-9a9e-ab1aa82d4d21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:28:59.637706 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:28:59.637684    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baf1982-c332-413c-9a9e-ab1aa82d4d21-maas-api-service-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:29:00.339257 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:00.339216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx" event={"ID":"3baf1982-c332-413c-9a9e-ab1aa82d4d21","Type":"ContainerDied","Data":"9962a07de3a1e5c135f519373a2156a001d0cca272fff52e9737037c9881f025"}
Jul 23 16:29:00.339407 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:00.339270    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9962a07de3a1e5c135f519373a2156a001d0cca272fff52e9737037c9881f025"
Jul 23 16:29:00.339407 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:00.339227    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"
Jul 23 16:29:16.508467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:16.508430    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"]
Jul 23 16:29:16.511088 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:16.511064    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bf4b9a-zswmx"]
Jul 23 16:29:16.771913 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:16.771833    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:29:16.810092 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:16.810060    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3baf1982-c332-413c-9a9e-ab1aa82d4d21" path="/var/lib/kubelet/pods/3baf1982-c332-413c-9a9e-ab1aa82d4d21/volumes"
Jul 23 16:29:34.715595 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:34.715557    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:29:49.323852 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:49.323808    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:29:57.120149 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.120118    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"]
Jul 23 16:29:57.120518 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.120444    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3baf1982-c332-413c-9a9e-ab1aa82d4d21" containerName="revoke-keys"
Jul 23 16:29:57.120518 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.120454    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3baf1982-c332-413c-9a9e-ab1aa82d4d21" containerName="revoke-keys"
Jul 23 16:29:57.120518 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.120511    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3baf1982-c332-413c-9a9e-ab1aa82d4d21" containerName="revoke-keys"
Jul 23 16:29:57.123537 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.123518    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.126061 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.126040    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-759481\"/\"auth-test-model-759481-kserve-self-signed-certs\""
Jul 23 16:29:57.126978 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.126963    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-759481\"/\"openshift-service-ca.crt\""
Jul 23 16:29:57.127103 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.127081    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-759481\"/\"default-dockercfg-s7m5b\""
Jul 23 16:29:57.127152 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.127091    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-759481\"/\"kube-root-ca.crt\""
Jul 23 16:29:57.134495 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.134473    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"]
Jul 23 16:29:57.254848 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.254809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-dshm\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.254848 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.254847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tmp-dir\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.255108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.254872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tls-certs\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.255108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.254917    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rghld\" (UniqueName: \"kubernetes.io/projected/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-kube-api-access-rghld\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.255108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.255022    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-home\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.255254 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.255150    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-model-cache\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356029 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.355990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-home\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356232 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.356057    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-model-cache\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356232 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.356193    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-dshm\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356350 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.356237    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tmp-dir\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356350 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.356271    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tls-certs\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356350 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.356297    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rghld\" (UniqueName: \"kubernetes.io/projected/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-kube-api-access-rghld\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356521 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.356479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-home\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356579 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.356535    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-model-cache\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.356791 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.356717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tmp-dir\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.358563 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.358538    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-dshm\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.358836 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.358820    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tls-certs\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.365004 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.364984    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rghld\" (UniqueName: \"kubernetes.io/projected/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-kube-api-access-rghld\") pod \"auth-test-model-759481-kserve-785b858ff5-nzmvq\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") " pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.433163 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.433126    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:29:57.568717 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.568692    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"]
Jul 23 16:29:57.571093 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:29:57.571063    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8a3716c_d04b_457c_a0cf_82b0e4ce98e7.slice/crio-e8e792c10a50101e14d4cbc775ee7f5c998e52b861a3d68a61c75bf8e5b9e525 WatchSource:0}: Error finding container e8e792c10a50101e14d4cbc775ee7f5c998e52b861a3d68a61c75bf8e5b9e525: Status 404 returned error can't find the container with id e8e792c10a50101e14d4cbc775ee7f5c998e52b861a3d68a61c75bf8e5b9e525
Jul 23 16:29:57.597343 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:57.597312    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq" event={"ID":"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7","Type":"ContainerStarted","Data":"e8e792c10a50101e14d4cbc775ee7f5c998e52b861a3d68a61c75bf8e5b9e525"}
Jul 23 16:29:58.602768 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:58.602698    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq" event={"ID":"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7","Type":"ContainerStarted","Data":"29fe03927c9963b39af54cae129eab943704f5f6c49a4da52602cee8e7881d5a"}
Jul 23 16:29:58.619082 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:29:58.619031    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq" podStartSLOduration=1.619014809 podStartE2EDuration="1.619014809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:29:58.618100369 +0000 UTC m=+2486.243634845" watchObservedRunningTime="2026-07-23 16:29:58.619014809 +0000 UTC m=+2486.244549284"
Jul 23 16:30:00.140555 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.140516    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"]
Jul 23 16:30:00.143967 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.143949    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l"]
Jul 23 16:30:00.144104 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.144085    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"
Jul 23 16:30:00.146703 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.146677    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6jwk\""
Jul 23 16:30:00.147137 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.147116    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"]
Jul 23 16:30:00.147263 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.147244    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l"
Jul 23 16:30:00.150224 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.150201    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"
Jul 23 16:30:00.151768 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.151722    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l"]
Jul 23 16:30:00.155735 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.155713    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"]
Jul 23 16:30:00.159164 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.159140    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"
Jul 23 16:30:00.167994 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.167973    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l"
Jul 23 16:30:00.168980 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.168959    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"]
Jul 23 16:30:00.201648 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.201623    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"
Jul 23 16:30:00.326700 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.326658    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l"]
Jul 23 16:30:00.330183 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:30:00.330151    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb048d558_3dca_486f_b078_e1766ba115b3.slice/crio-517cd7fac2519e20ae2408e94661d05b3101acc574ad5c377d5f931d3451a3ce WatchSource:0}: Error finding container 517cd7fac2519e20ae2408e94661d05b3101acc574ad5c377d5f931d3451a3ce: Status 404 returned error can't find the container with id 517cd7fac2519e20ae2408e94661d05b3101acc574ad5c377d5f931d3451a3ce
Jul 23 16:30:00.375560 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.375531    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"]
Jul 23 16:30:00.377073 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:30:00.377047    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ce5510_e04b_48dc_8cfe_a01819373401.slice/crio-5ac3397f2aad4ed9b49bafd599414e24713a9f6c36a41eca442829eda7ca17cf WatchSource:0}: Error finding container 5ac3397f2aad4ed9b49bafd599414e24713a9f6c36a41eca442829eda7ca17cf: Status 404 returned error can't find the container with id 5ac3397f2aad4ed9b49bafd599414e24713a9f6c36a41eca442829eda7ca17cf
Jul 23 16:30:00.556958 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.556925    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"]
Jul 23 16:30:00.566889 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:30:00.566858    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a38d7f_3441_4fd7_8e9a_1f618561aff7.slice/crio-ddb09161d8ac34a5ce9b47a0938b4ec392af2df4f186ddc591b793173b138342 WatchSource:0}: Error finding container ddb09161d8ac34a5ce9b47a0938b4ec392af2df4f186ddc591b793173b138342: Status 404 returned error can't find the container with id ddb09161d8ac34a5ce9b47a0938b4ec392af2df4f186ddc591b793173b138342
Jul 23 16:30:00.613594 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.613563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp" event={"ID":"88ce5510-e04b-48dc-8cfe-a01819373401","Type":"ContainerStarted","Data":"5ac3397f2aad4ed9b49bafd599414e24713a9f6c36a41eca442829eda7ca17cf"}
Jul 23 16:30:00.614636 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.614614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l" event={"ID":"b048d558-3dca-486f-b078-e1766ba115b3","Type":"ContainerStarted","Data":"517cd7fac2519e20ae2408e94661d05b3101acc574ad5c377d5f931d3451a3ce"}
Jul 23 16:30:00.615711 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:00.615685    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b" event={"ID":"94a38d7f-3441-4fd7-8e9a-1f618561aff7","Type":"ContainerStarted","Data":"ddb09161d8ac34a5ce9b47a0938b4ec392af2df4f186ddc591b793173b138342"}
Jul 23 16:30:01.514095 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:01.514060    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:30:01.622260 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:01.622174    2555 generic.go:356] "Generic (PLEG): container finished" podID="88ce5510-e04b-48dc-8cfe-a01819373401" containerID="21dad5c31c93194e13a8a093b66c74c043c8e383f361bcbfce15d689497498b4" exitCode=0
Jul 23 16:30:01.622260 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:01.622252    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp" event={"ID":"88ce5510-e04b-48dc-8cfe-a01819373401","Type":"ContainerDied","Data":"21dad5c31c93194e13a8a093b66c74c043c8e383f361bcbfce15d689497498b4"}
Jul 23 16:30:01.623720 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:01.623695    2555 generic.go:356] "Generic (PLEG): container finished" podID="b048d558-3dca-486f-b078-e1766ba115b3" containerID="0dd6433db874ca0e3d1611218599a81aeb1246d9b081cb0e8ca5d4fa7142d85f" exitCode=0
Jul 23 16:30:01.623857 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:01.623779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l" event={"ID":"b048d558-3dca-486f-b078-e1766ba115b3","Type":"ContainerDied","Data":"0dd6433db874ca0e3d1611218599a81aeb1246d9b081cb0e8ca5d4fa7142d85f"}
Jul 23 16:30:01.625401 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:01.625378    2555 generic.go:356] "Generic (PLEG): container finished" podID="94a38d7f-3441-4fd7-8e9a-1f618561aff7" containerID="9dc3a4238bc785ccf9582617a3a15c3ab1d389d569397b1d02df406f46ff02f1" exitCode=0
Jul 23 16:30:01.625503 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:01.625412    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b" event={"ID":"94a38d7f-3441-4fd7-8e9a-1f618561aff7","Type":"ContainerDied","Data":"9dc3a4238bc785ccf9582617a3a15c3ab1d389d569397b1d02df406f46ff02f1"}
Jul 23 16:30:02.773583 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:02.773553    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"
Jul 23 16:30:02.827571 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:02.827552    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l"
Jul 23 16:30:02.831568 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:02.831544    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"
Jul 23 16:30:03.636216 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.636184    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp" event={"ID":"88ce5510-e04b-48dc-8cfe-a01819373401","Type":"ContainerDied","Data":"5ac3397f2aad4ed9b49bafd599414e24713a9f6c36a41eca442829eda7ca17cf"}
Jul 23 16:30:03.636216 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.636204    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"
Jul 23 16:30:03.636427 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.636262    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ac3397f2aad4ed9b49bafd599414e24713a9f6c36a41eca442829eda7ca17cf"
Jul 23 16:30:03.637464 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.637448    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l"
Jul 23 16:30:03.637563 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.637447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747070-mkh6l" event={"ID":"b048d558-3dca-486f-b078-e1766ba115b3","Type":"ContainerDied","Data":"517cd7fac2519e20ae2408e94661d05b3101acc574ad5c377d5f931d3451a3ce"}
Jul 23 16:30:03.637563 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.637559    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517cd7fac2519e20ae2408e94661d05b3101acc574ad5c377d5f931d3451a3ce"
Jul 23 16:30:03.638881 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.638859    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"
Jul 23 16:30:03.638881 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.638873    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b" event={"ID":"94a38d7f-3441-4fd7-8e9a-1f618561aff7","Type":"ContainerDied","Data":"ddb09161d8ac34a5ce9b47a0938b4ec392af2df4f186ddc591b793173b138342"}
Jul 23 16:30:03.639030 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:03.638911    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddb09161d8ac34a5ce9b47a0938b4ec392af2df4f186ddc591b793173b138342"
Jul 23 16:30:07.434303 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:07.434267    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:30:07.434303 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:07.434315    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:30:07.446709 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:07.446683    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:30:07.668790 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:07.668760    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:30:34.014520 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:30:34.014427    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:31:04.909693 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:31:04.909655    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:31:22.214850 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:31:22.214814    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:31:38.811788 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:31:38.811731    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:31:58.305266 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:31:58.305181    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:32:15.518813 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:15.518769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:32:33.817071 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:33.817035    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:32:50.266849 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:50.266816    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"]
Jul 23 16:32:54.515129 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:54.515080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:32:59.237764 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.237717    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"]
Jul 23 16:32:59.238139 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238078    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b048d558-3dca-486f-b078-e1766ba115b3" containerName="cleanup"
Jul 23 16:32:59.238139 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238090    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b048d558-3dca-486f-b078-e1766ba115b3" containerName="cleanup"
Jul 23 16:32:59.238139 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238099    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a38d7f-3441-4fd7-8e9a-1f618561aff7" containerName="cleanup"
Jul 23 16:32:59.238139 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238104    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a38d7f-3441-4fd7-8e9a-1f618561aff7" containerName="cleanup"
Jul 23 16:32:59.238139 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238113    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ce5510-e04b-48dc-8cfe-a01819373401" containerName="cleanup"
Jul 23 16:32:59.238139 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238119    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ce5510-e04b-48dc-8cfe-a01819373401" containerName="cleanup"
Jul 23 16:32:59.238336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238174    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a38d7f-3441-4fd7-8e9a-1f618561aff7" containerName="cleanup"
Jul 23 16:32:59.238336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238181    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b048d558-3dca-486f-b078-e1766ba115b3" containerName="cleanup"
Jul 23 16:32:59.238336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.238189    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ce5510-e04b-48dc-8cfe-a01819373401" containerName="cleanup"
Jul 23 16:32:59.241154 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.241136    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.243915 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.243888    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-759481\"/\"sub-test-model-759481-kserve-self-signed-certs\""
Jul 23 16:32:59.253526 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.253499    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"]
Jul 23 16:32:59.268083 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.268055    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-model-cache\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.268205 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.268092    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tmp-dir\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.268205 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.268112    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-dshm\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.268205 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.268184    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmdk\" (UniqueName: \"kubernetes.io/projected/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-kube-api-access-grmdk\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.268312 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.268220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tls-certs\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.268312 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.268261    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-home\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.368744 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.368706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tls-certs\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.368950 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.368786    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-home\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.368950 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.368822    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-model-cache\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.368950 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.368840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tmp-dir\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.368950 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.368859    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-dshm\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.368950 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.368889    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmdk\" (UniqueName: \"kubernetes.io/projected/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-kube-api-access-grmdk\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.369357 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.369334    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-home\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.369418 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.369344    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-model-cache\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.369418 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.369405    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tmp-dir\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.371146 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.371124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-dshm\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.371290 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.371273    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tls-certs\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.377379 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.377355    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmdk\" (UniqueName: \"kubernetes.io/projected/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-kube-api-access-grmdk\") pod \"sub-test-model-759481-kserve-588c6f75d7-dcfk6\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") " pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.550577 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.550480    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:32:59.709253 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:32:59.709232    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"]
Jul 23 16:32:59.711906 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:32:59.711877    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56ea61ff_8d1f_4c4b_b940_2d9bc41b00c1.slice/crio-a446039238fb49d25e489a7c0cac04ea8ddd5867cba2d719e6973eb9d80baa7c WatchSource:0}: Error finding container a446039238fb49d25e489a7c0cac04ea8ddd5867cba2d719e6973eb9d80baa7c: Status 404 returned error can't find the container with id a446039238fb49d25e489a7c0cac04ea8ddd5867cba2d719e6973eb9d80baa7c
Jul 23 16:33:00.478084 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:00.478050    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6" event={"ID":"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1","Type":"ContainerStarted","Data":"108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe"}
Jul 23 16:33:00.478084 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:00.478097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6" event={"ID":"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1","Type":"ContainerStarted","Data":"a446039238fb49d25e489a7c0cac04ea8ddd5867cba2d719e6973eb9d80baa7c"}
Jul 23 16:33:05.287061 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.287023    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq" podUID="a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" containerName="main" containerID="cri-o://29fe03927c9963b39af54cae129eab943704f5f6c49a4da52602cee8e7881d5a" gracePeriod=45
Jul 23 16:33:05.502974 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.502943    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" containerID="29fe03927c9963b39af54cae129eab943704f5f6c49a4da52602cee8e7881d5a" exitCode=0
Jul 23 16:33:05.503141 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.503013    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq" event={"ID":"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7","Type":"ContainerDied","Data":"29fe03927c9963b39af54cae129eab943704f5f6c49a4da52602cee8e7881d5a"}
Jul 23 16:33:05.536832 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.536809    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:33:05.554823 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.554110    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6" podStartSLOduration=6.554090606 podStartE2EDuration="6.554090606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:33:00.496196315 +0000 UTC m=+2668.121730790" watchObservedRunningTime="2026-07-23 16:33:05.554090606 +0000 UTC m=+2673.179625091"
Jul 23 16:33:05.622128 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622097    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-home\") pod \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") "
Jul 23 16:33:05.622325 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622176    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rghld\" (UniqueName: \"kubernetes.io/projected/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-kube-api-access-rghld\") pod \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") "
Jul 23 16:33:05.622325 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622208    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-dshm\") pod \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") "
Jul 23 16:33:05.622325 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622260    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-model-cache\") pod \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") "
Jul 23 16:33:05.622504 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622376    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tls-certs\") pod \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") "
Jul 23 16:33:05.622504 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622416    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tmp-dir\") pod \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\" (UID: \"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7\") "
Jul 23 16:33:05.622504 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622402    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-home" (OuterVolumeSpecName: "home") pod "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" (UID: "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:33:05.622504 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622448    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" (UID: "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:33:05.622703 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622673    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:33:05.622703 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622687    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:33:05.622833 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.622716    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" (UID: "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:33:05.624366 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.624345    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-dshm" (OuterVolumeSpecName: "dshm") pod "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" (UID: "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:33:05.624468 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.624446    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" (UID: "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 16:33:05.624580 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.624562    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-kube-api-access-rghld" (OuterVolumeSpecName: "kube-api-access-rghld") pod "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" (UID: "a8a3716c-d04b-457c-a0cf-82b0e4ce98e7"). InnerVolumeSpecName "kube-api-access-rghld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:33:05.723630 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.723589    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rghld\" (UniqueName: \"kubernetes.io/projected/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-kube-api-access-rghld\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:33:05.723630 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.723621    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:33:05.723630 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.723635    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:33:05.723928 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:05.723647    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:33:06.509388 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:06.509354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq" event={"ID":"a8a3716c-d04b-457c-a0cf-82b0e4ce98e7","Type":"ContainerDied","Data":"e8e792c10a50101e14d4cbc775ee7f5c998e52b861a3d68a61c75bf8e5b9e525"}
Jul 23 16:33:06.509388 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:06.509382    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"
Jul 23 16:33:06.509886 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:06.509413    2555 scope.go:117] "RemoveContainer" containerID="29fe03927c9963b39af54cae129eab943704f5f6c49a4da52602cee8e7881d5a"
Jul 23 16:33:06.548137 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:06.548108    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"]
Jul 23 16:33:06.574018 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:06.573986    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-759481/auth-test-model-759481-kserve-785b858ff5-nzmvq"]
Jul 23 16:33:06.810310 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:06.810216    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" path="/var/lib/kubelet/pods/a8a3716c-d04b-457c-a0cf-82b0e4ce98e7/volumes"
Jul 23 16:33:08.610371 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:08.610339    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:33:09.551098 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:09.551065    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:33:09.551256 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:09.551111    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:33:09.563702 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:09.563668    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:33:10.540887 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:10.540856    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:33:25.924638 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.924560    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"]
Jul 23 16:33:25.925108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.924940    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" containerName="main"
Jul 23 16:33:25.925108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.924954    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" containerName="main"
Jul 23 16:33:25.925108 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.925025    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a3716c-d04b-457c-a0cf-82b0e4ce98e7" containerName="main"
Jul 23 16:33:25.927978 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.927960    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:25.931550 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.931526    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3a6a71\"/\"openshift-service-ca.crt\""
Jul 23 16:33:25.931671 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.931526    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3a6a71\"/\"sub-test-model-3a6a71-kserve-self-signed-certs\""
Jul 23 16:33:25.931719 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.931703    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3a6a71\"/\"default-dockercfg-jdkvj\""
Jul 23 16:33:25.931807 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.931786    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3a6a71\"/\"kube-root-ca.crt\""
Jul 23 16:33:25.938691 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:25.938665    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"]
Jul 23 16:33:26.013498 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.013468    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-model-cache\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.013672 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.013523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-home\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.013672 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.013579    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxrt\" (UniqueName: \"kubernetes.io/projected/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-kube-api-access-txxrt\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.013672 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.013617    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tls-certs\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.013672 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.013640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-dshm\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.013848 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.013723    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tmp-dir\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.114806 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.114745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-home\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.114806 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.114814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxrt\" (UniqueName: \"kubernetes.io/projected/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-kube-api-access-txxrt\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.115068 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.114833    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tls-certs\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.115068 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.114857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-dshm\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.115068 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.114912    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tmp-dir\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.115068 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.114965    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-model-cache\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.115292 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.115260    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-home\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.115337 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.115308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tmp-dir\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.115372 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.115333    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-model-cache\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.117239 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.117205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-dshm\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.117463 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.117447    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tls-certs\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.125796 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.125774    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxrt\" (UniqueName: \"kubernetes.io/projected/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-kube-api-access-txxrt\") pod \"sub-test-model-3a6a71-kserve-7885d7c785-m2nd6\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") " pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.237255 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.237173    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:26.377216 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.375370    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"]
Jul 23 16:33:26.605522 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.605429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6" event={"ID":"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254","Type":"ContainerStarted","Data":"32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951"}
Jul 23 16:33:26.605522 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.605471    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6" event={"ID":"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254","Type":"ContainerStarted","Data":"be4c0dea577dec21ef0fcd5bba74a57145ee5c5832f873d2215024ff2db27237"}
Jul 23 16:33:26.629812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:26.629741    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6" podStartSLOduration=1.629725755 podStartE2EDuration="1.629725755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:33:26.628541897 +0000 UTC m=+2694.254076372" watchObservedRunningTime="2026-07-23 16:33:26.629725755 +0000 UTC m=+2694.255260273"
Jul 23 16:33:29.922912 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:29.922875    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:33:32.912347 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:32.912314    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:33:32.915944 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:32.915917    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:33:36.237363 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:36.237314    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:36.237908 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:36.237383    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:36.250111 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:36.250082    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:36.664711 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:36.664676    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:33:44.322034 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:33:44.321996    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:34:00.430245 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:00.430209    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:34:06.508417 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:06.508387    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"]
Jul 23 16:34:07.331415 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:07.331384    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"]
Jul 23 16:34:20.706098 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:20.706063    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:34:21.527971 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.527927    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6" podUID="56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" containerName="main" containerID="cri-o://108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe" gracePeriod=45
Jul 23 16:34:21.784044 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.783976    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:34:21.870011 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.869978    2555 generic.go:356] "Generic (PLEG): container finished" podID="56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" containerID="108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe" exitCode=0
Jul 23 16:34:21.870180 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.870015    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"
Jul 23 16:34:21.870180 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.870021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6" event={"ID":"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1","Type":"ContainerDied","Data":"108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe"}
Jul 23 16:34:21.870180 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.870058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6" event={"ID":"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1","Type":"ContainerDied","Data":"a446039238fb49d25e489a7c0cac04ea8ddd5867cba2d719e6973eb9d80baa7c"}
Jul 23 16:34:21.870180 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.870077    2555 scope.go:117] "RemoveContainer" containerID="108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe"
Jul 23 16:34:21.877730 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.877704    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-dshm\") pod \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") "
Jul 23 16:34:21.877898 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.877734    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tmp-dir\") pod \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") "
Jul 23 16:34:21.877898 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.877803    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tls-certs\") pod \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") "
Jul 23 16:34:21.877898 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.877855    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-home\") pod \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") "
Jul 23 16:34:21.878065 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.877982    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-model-cache\") pod \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") "
Jul 23 16:34:21.878065 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.878020    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grmdk\" (UniqueName: \"kubernetes.io/projected/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-kube-api-access-grmdk\") pod \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\" (UID: \"56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1\") "
Jul 23 16:34:21.878196 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.878098    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" (UID: "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:34:21.878269 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.878243    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-home" (OuterVolumeSpecName: "home") pod "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" (UID: "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:34:21.878331 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.878266    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" (UID: "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:34:21.878445 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.878401    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:21.878445 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.878427    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:21.878445 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.878443    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:21.878871 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.878849    2555 scope.go:117] "RemoveContainer" containerID="108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe"
Jul 23 16:34:21.879223 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:34:21.879198    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe\": container with ID starting with 108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe not found: ID does not exist" containerID="108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe"
Jul 23 16:34:21.879309 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.879236    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe"} err="failed to get container status \"108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe\": rpc error: code = NotFound desc = could not find container \"108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe\": container with ID starting with 108e1a1df585b06555d41c5c1b827689d66a196a9560f80bf862c2ed504700fe not found: ID does not exist"
Jul 23 16:34:21.880006 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.879983    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-dshm" (OuterVolumeSpecName: "dshm") pod "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" (UID: "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:34:21.880106 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.880087    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" (UID: "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 16:34:21.880106 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.880095    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-kube-api-access-grmdk" (OuterVolumeSpecName: "kube-api-access-grmdk") pod "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" (UID: "56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1"). InnerVolumeSpecName "kube-api-access-grmdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:34:21.979798 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.979720    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:21.979798 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.979794    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grmdk\" (UniqueName: \"kubernetes.io/projected/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-kube-api-access-grmdk\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:21.979798 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:21.979805    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:22.197909 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.197875    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"]
Jul 23 16:34:22.204976 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.204948    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-759481/sub-test-model-759481-kserve-588c6f75d7-dcfk6"]
Jul 23 16:34:22.352645 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.352591    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6" podUID="3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" containerName="main" containerID="cri-o://32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951" gracePeriod=45
Jul 23 16:34:22.610326 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.610293    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:34:22.687152 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687115    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tmp-dir\") pod \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") "
Jul 23 16:34:22.687152 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687153    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-model-cache\") pod \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") "
Jul 23 16:34:22.687409 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687241    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-home\") pod \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") "
Jul 23 16:34:22.687409 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687264    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tls-certs\") pod \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") "
Jul 23 16:34:22.687409 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687310    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txxrt\" (UniqueName: \"kubernetes.io/projected/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-kube-api-access-txxrt\") pod \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") "
Jul 23 16:34:22.687409 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687332    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-dshm\") pod \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\" (UID: \"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254\") "
Jul 23 16:34:22.687626 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687463    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" (UID: "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:34:22.687626 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687470    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" (UID: "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:34:22.687626 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.687543    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-home" (OuterVolumeSpecName: "home") pod "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" (UID: "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:34:22.689696 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.689669    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-kube-api-access-txxrt" (OuterVolumeSpecName: "kube-api-access-txxrt") pod "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" (UID: "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254"). InnerVolumeSpecName "kube-api-access-txxrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:34:22.689696 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.689672    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" (UID: "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 16:34:22.689866 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.689675    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-dshm" (OuterVolumeSpecName: "dshm") pod "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" (UID: "3d420a5c-bad6-4cd0-852c-e6cb9a0b0254"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:34:22.788812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.788690    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:22.788812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.788727    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:22.788812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.788737    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txxrt\" (UniqueName: \"kubernetes.io/projected/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-kube-api-access-txxrt\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:22.788812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.788772    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:22.788812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.788781    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:22.788812 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.788789    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:34:22.809305 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.809277    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" path="/var/lib/kubelet/pods/56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1/volumes"
Jul 23 16:34:22.876935 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.876901    2555 generic.go:356] "Generic (PLEG): container finished" podID="3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" containerID="32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951" exitCode=0
Jul 23 16:34:22.877124 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.876946    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"
Jul 23 16:34:22.877124 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.876967    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6" event={"ID":"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254","Type":"ContainerDied","Data":"32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951"}
Jul 23 16:34:22.877124 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.876996    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6" event={"ID":"3d420a5c-bad6-4cd0-852c-e6cb9a0b0254","Type":"ContainerDied","Data":"be4c0dea577dec21ef0fcd5bba74a57145ee5c5832f873d2215024ff2db27237"}
Jul 23 16:34:22.877124 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.877013    2555 scope.go:117] "RemoveContainer" containerID="32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951"
Jul 23 16:34:22.885560 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.885536    2555 scope.go:117] "RemoveContainer" containerID="32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951"
Jul 23 16:34:22.885822 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:34:22.885801    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951\": container with ID starting with 32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951 not found: ID does not exist" containerID="32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951"
Jul 23 16:34:22.885887 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.885831    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951"} err="failed to get container status \"32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951\": rpc error: code = NotFound desc = could not find container \"32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951\": container with ID starting with 32ecc8405212b13dbd7a7d285e687faedbe3b4c4e7848b4870322248cdea5951 not found: ID does not exist"
Jul 23 16:34:22.894406 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.894378    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"]
Jul 23 16:34:22.898440 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:22.898416    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/sub-test-model-3a6a71-kserve-7885d7c785-m2nd6"]
Jul 23 16:34:24.810071 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:24.810039    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" path="/var/lib/kubelet/pods/3d420a5c-bad6-4cd0-852c-e6cb9a0b0254/volumes"
Jul 23 16:34:33.033228 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:33.033197    2555 scope.go:117] "RemoveContainer" containerID="ab351eb764b4d8462c6195484288742ec5beedbf5df3cd3804204e7ba842ed9c"
Jul 23 16:34:52.712445 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.712405    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"]
Jul 23 16:34:52.712993 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.712968    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" containerName="main"
Jul 23 16:34:52.712993 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.712993    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" containerName="main"
Jul 23 16:34:52.713200 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.713005    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" containerName="main"
Jul 23 16:34:52.713200 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.713013    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" containerName="main"
Jul 23 16:34:52.713200 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.713130    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ea61ff-8d1f-4c4b-b940-2d9bc41b00c1" containerName="main"
Jul 23 16:34:52.713200 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.713143    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d420a5c-bad6-4cd0-852c-e6cb9a0b0254" containerName="main"
Jul 23 16:34:52.716133 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.716111    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.719321 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.719292    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3a6a71\"/\"openshift-service-ca.crt\""
Jul 23 16:34:52.719450 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.719354    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3a6a71\"/\"rate-test-model-3a6a71-kserve-self-signed-certs\""
Jul 23 16:34:52.719564 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.719546    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3a6a71\"/\"default-dockercfg-jdkvj\""
Jul 23 16:34:52.719630 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.719577    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3a6a71\"/\"kube-root-ca.crt\""
Jul 23 16:34:52.725361 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.725340    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"]
Jul 23 16:34:52.758290 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.758256    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-model-cache\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.758443 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.758307    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-home\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.758443 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.758383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-tmp-dir\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.758443 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.758425    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-dshm\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.758569 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.758456    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglkj\" (UniqueName: \"kubernetes.io/projected/5f305559-060c-4bfa-8694-0a6d8baad573-kube-api-access-pglkj\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.758569 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.758492    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f305559-060c-4bfa-8694-0a6d8baad573-tls-certs\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.859938 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.859907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-tmp-dir\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.860118 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.859952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-dshm\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.860118 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.859988    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglkj\" (UniqueName: \"kubernetes.io/projected/5f305559-060c-4bfa-8694-0a6d8baad573-kube-api-access-pglkj\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.860118 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.860018    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f305559-060c-4bfa-8694-0a6d8baad573-tls-certs\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.860118 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.860064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-model-cache\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.860324 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.860206    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-home\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.860371 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.860351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-tmp-dir\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.860482 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.860445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-model-cache\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.860530 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.860512    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-home\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.862251 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.862231    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-dshm\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.862536 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.862519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f305559-060c-4bfa-8694-0a6d8baad573-tls-certs\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:52.868623 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:52.868596    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglkj\" (UniqueName: \"kubernetes.io/projected/5f305559-060c-4bfa-8694-0a6d8baad573-kube-api-access-pglkj\") pod \"rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") " pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:53.025872 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:53.025777    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:34:53.161423 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:53.161401    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"]
Jul 23 16:34:53.163042 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:34:53.163001    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f305559_060c_4bfa_8694_0a6d8baad573.slice/crio-f9037614578d25cc4e8d9f0a5f7d30908396cc181a07542b9aed504aa87044ae WatchSource:0}: Error finding container f9037614578d25cc4e8d9f0a5f7d30908396cc181a07542b9aed504aa87044ae: Status 404 returned error can't find the container with id f9037614578d25cc4e8d9f0a5f7d30908396cc181a07542b9aed504aa87044ae
Jul 23 16:34:54.021078 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:54.021037    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n" event={"ID":"5f305559-060c-4bfa-8694-0a6d8baad573","Type":"ContainerStarted","Data":"0537f160aa0be3514400f36049c4975a3e0844e58de3eb86902c2bc82f202795"}
Jul 23 16:34:54.021473 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:54.021087    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n" event={"ID":"5f305559-060c-4bfa-8694-0a6d8baad573","Type":"ContainerStarted","Data":"f9037614578d25cc4e8d9f0a5f7d30908396cc181a07542b9aed504aa87044ae"}
Jul 23 16:34:54.039010 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:54.038961    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n" podStartSLOduration=2.038944692 podStartE2EDuration="2.038944692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 16:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:34:54.036949157 +0000 UTC m=+2781.662483634" watchObservedRunningTime="2026-07-23 16:34:54.038944692 +0000 UTC m=+2781.664479229"
Jul 23 16:34:56.117508 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:34:56.117421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:35:03.026706 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:03.026660    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:35:03.027128 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:03.026769    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:35:03.039442 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:03.039401    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:35:03.073336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:03.073299    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:35:10.617962 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:10.617924    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:35:25.607549 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:25.607507    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:35:33.059904 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:33.059869    2555 scope.go:117] "RemoveContainer" containerID="5c4024b0c9e2e1a547ff968824a469277756421980b743b464a9e0d32a998320"
Jul 23 16:35:33.121946 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:33.121907    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"]
Jul 23 16:35:34.170367 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.170335    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7461-140507304202928"]
Jul 23 16:35:34.175444 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.175420    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304202928"
Jul 23 16:35:34.178193 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.178162    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5x8p\""
Jul 23 16:35:34.180366 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.180341    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304202928"]
Jul 23 16:35:34.226720 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.226688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlq4\" (UniqueName: \"kubernetes.io/projected/f9ec4541-ef7c-4843-816b-55aaf6d1e4ad-kube-api-access-9mlq4\") pod \"test-curl-7461-140507304202928\" (UID: \"f9ec4541-ef7c-4843-816b-55aaf6d1e4ad\") " pod="opendatahub/test-curl-7461-140507304202928"
Jul 23 16:35:34.328141 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.328099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlq4\" (UniqueName: \"kubernetes.io/projected/f9ec4541-ef7c-4843-816b-55aaf6d1e4ad-kube-api-access-9mlq4\") pod \"test-curl-7461-140507304202928\" (UID: \"f9ec4541-ef7c-4843-816b-55aaf6d1e4ad\") " pod="opendatahub/test-curl-7461-140507304202928"
Jul 23 16:35:34.337172 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.337144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlq4\" (UniqueName: \"kubernetes.io/projected/f9ec4541-ef7c-4843-816b-55aaf6d1e4ad-kube-api-access-9mlq4\") pod \"test-curl-7461-140507304202928\" (UID: \"f9ec4541-ef7c-4843-816b-55aaf6d1e4ad\") " pod="opendatahub/test-curl-7461-140507304202928"
Jul 23 16:35:34.486063 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.485981    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304202928"
Jul 23 16:35:34.627717 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.622967    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304202928"]
Jul 23 16:35:34.629502 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:35:34.629474    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9ec4541_ef7c_4843_816b_55aaf6d1e4ad.slice/crio-adf31d36ed1bb5906d0571c477891ddfd63ed1cf877ba8d9ea636983058cf265 WatchSource:0}: Error finding container adf31d36ed1bb5906d0571c477891ddfd63ed1cf877ba8d9ea636983058cf265: Status 404 returned error can't find the container with id adf31d36ed1bb5906d0571c477891ddfd63ed1cf877ba8d9ea636983058cf265
Jul 23 16:35:34.631210 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:34.631193    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 16:35:35.210764 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:35.210700    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304202928" event={"ID":"f9ec4541-ef7c-4843-816b-55aaf6d1e4ad","Type":"ContainerStarted","Data":"adf31d36ed1bb5906d0571c477891ddfd63ed1cf877ba8d9ea636983058cf265"}
Jul 23 16:35:36.216982 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:36.216944    2555 generic.go:356] "Generic (PLEG): container finished" podID="f9ec4541-ef7c-4843-816b-55aaf6d1e4ad" containerID="5a3d9eafdaf24af2b5abb30189dd4a4117c2008b667e9409ca5b570abb53a121" exitCode=0
Jul 23 16:35:36.217351 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:36.217013    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304202928" event={"ID":"f9ec4541-ef7c-4843-816b-55aaf6d1e4ad","Type":"ContainerDied","Data":"5a3d9eafdaf24af2b5abb30189dd4a4117c2008b667e9409ca5b570abb53a121"}
Jul 23 16:35:37.360332 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.360306    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304202928"
Jul 23 16:35:37.396609 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.396573    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7461-140507304202928_f9ec4541-ef7c-4843-816b-55aaf6d1e4ad/test-curl-7461-140507304202928/0.log"
Jul 23 16:35:37.419399 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.419362    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7461-140507304202928"]
Jul 23 16:35:37.423197 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.423173    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7461-140507304202928"]
Jul 23 16:35:37.457207 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.457178    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mlq4\" (UniqueName: \"kubernetes.io/projected/f9ec4541-ef7c-4843-816b-55aaf6d1e4ad-kube-api-access-9mlq4\") pod \"f9ec4541-ef7c-4843-816b-55aaf6d1e4ad\" (UID: \"f9ec4541-ef7c-4843-816b-55aaf6d1e4ad\") "
Jul 23 16:35:37.459368 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.459342    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ec4541-ef7c-4843-816b-55aaf6d1e4ad-kube-api-access-9mlq4" (OuterVolumeSpecName: "kube-api-access-9mlq4") pod "f9ec4541-ef7c-4843-816b-55aaf6d1e4ad" (UID: "f9ec4541-ef7c-4843-816b-55aaf6d1e4ad"). InnerVolumeSpecName "kube-api-access-9mlq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:35:37.552875 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.552786    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7461-140507304161584"]
Jul 23 16:35:37.553135 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.553123    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ec4541-ef7c-4843-816b-55aaf6d1e4ad" containerName="test-curl-7461-140507304202928"
Jul 23 16:35:37.553178 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.553136    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ec4541-ef7c-4843-816b-55aaf6d1e4ad" containerName="test-curl-7461-140507304202928"
Jul 23 16:35:37.553225 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.553216    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ec4541-ef7c-4843-816b-55aaf6d1e4ad" containerName="test-curl-7461-140507304202928"
Jul 23 16:35:37.556118 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.556102    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304161584"
Jul 23 16:35:37.558571 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.558547    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mlq4\" (UniqueName: \"kubernetes.io/projected/f9ec4541-ef7c-4843-816b-55aaf6d1e4ad-kube-api-access-9mlq4\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:37.561892 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.561866    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304161584"]
Jul 23 16:35:37.659797 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.659744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rgcn\" (UniqueName: \"kubernetes.io/projected/0b2553bd-60de-4105-b7c3-a393f1e4cd51-kube-api-access-7rgcn\") pod \"test-curl-7461-140507304161584\" (UID: \"0b2553bd-60de-4105-b7c3-a393f1e4cd51\") " pod="opendatahub/test-curl-7461-140507304161584"
Jul 23 16:35:37.760602 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.760556    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rgcn\" (UniqueName: \"kubernetes.io/projected/0b2553bd-60de-4105-b7c3-a393f1e4cd51-kube-api-access-7rgcn\") pod \"test-curl-7461-140507304161584\" (UID: \"0b2553bd-60de-4105-b7c3-a393f1e4cd51\") " pod="opendatahub/test-curl-7461-140507304161584"
Jul 23 16:35:37.768605 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.768576    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rgcn\" (UniqueName: \"kubernetes.io/projected/0b2553bd-60de-4105-b7c3-a393f1e4cd51-kube-api-access-7rgcn\") pod \"test-curl-7461-140507304161584\" (UID: \"0b2553bd-60de-4105-b7c3-a393f1e4cd51\") " pod="opendatahub/test-curl-7461-140507304161584"
Jul 23 16:35:37.866634 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:37.866544    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304161584"
Jul 23 16:35:38.000396 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:38.000370    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304161584"]
Jul 23 16:35:38.002453 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:35:38.002425    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b2553bd_60de_4105_b7c3_a393f1e4cd51.slice/crio-ad4c12dc6f8bcb89035a68228597798bc16313c77e23c21c89a7d094422d2617 WatchSource:0}: Error finding container ad4c12dc6f8bcb89035a68228597798bc16313c77e23c21c89a7d094422d2617: Status 404 returned error can't find the container with id ad4c12dc6f8bcb89035a68228597798bc16313c77e23c21c89a7d094422d2617
Jul 23 16:35:38.227778 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:38.227721    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304161584" event={"ID":"0b2553bd-60de-4105-b7c3-a393f1e4cd51","Type":"ContainerStarted","Data":"ad4c12dc6f8bcb89035a68228597798bc16313c77e23c21c89a7d094422d2617"}
Jul 23 16:35:38.228913 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:38.228888    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304202928"
Jul 23 16:35:38.229049 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:38.228894    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adf31d36ed1bb5906d0571c477891ddfd63ed1cf877ba8d9ea636983058cf265"
Jul 23 16:35:38.239264 ip-10-0-128-121 kubenswrapper[2555]: E0723 16:35:38.239237    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7461-140507304202928\" is forbidden: User \"system:node:ip-10-0-128-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-121.ec2.internal' and this object" podUID="f9ec4541-ef7c-4843-816b-55aaf6d1e4ad" pod="opendatahub/test-curl-7461-140507304202928"
Jul 23 16:35:38.809991 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:38.809962    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ec4541-ef7c-4843-816b-55aaf6d1e4ad" path="/var/lib/kubelet/pods/f9ec4541-ef7c-4843-816b-55aaf6d1e4ad/volumes"
Jul 23 16:35:39.235686 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:39.235622    2555 generic.go:356] "Generic (PLEG): container finished" podID="0b2553bd-60de-4105-b7c3-a393f1e4cd51" containerID="c71faa822c49a075dea3a031fa99cbfefb0948fb321f63aaeb075af6ac194de4" exitCode=0
Jul 23 16:35:39.235892 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:39.235716    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304161584" event={"ID":"0b2553bd-60de-4105-b7c3-a393f1e4cd51","Type":"ContainerDied","Data":"c71faa822c49a075dea3a031fa99cbfefb0948fb321f63aaeb075af6ac194de4"}
Jul 23 16:35:40.306251 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.306201    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:35:40.382956 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.382924    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304161584"
Jul 23 16:35:40.402191 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.402149    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7461-140507304161584_0b2553bd-60de-4105-b7c3-a393f1e4cd51/test-curl-7461-140507304161584/0.log"
Jul 23 16:35:40.427421 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.427388    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7461-140507304161584"]
Jul 23 16:35:40.431240 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.431213    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7461-140507304161584"]
Jul 23 16:35:40.484825 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.484791    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rgcn\" (UniqueName: \"kubernetes.io/projected/0b2553bd-60de-4105-b7c3-a393f1e4cd51-kube-api-access-7rgcn\") pod \"0b2553bd-60de-4105-b7c3-a393f1e4cd51\" (UID: \"0b2553bd-60de-4105-b7c3-a393f1e4cd51\") "
Jul 23 16:35:40.486919 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.486885    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2553bd-60de-4105-b7c3-a393f1e4cd51-kube-api-access-7rgcn" (OuterVolumeSpecName: "kube-api-access-7rgcn") pod "0b2553bd-60de-4105-b7c3-a393f1e4cd51" (UID: "0b2553bd-60de-4105-b7c3-a393f1e4cd51"). InnerVolumeSpecName "kube-api-access-7rgcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:35:40.586310 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.586230    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rgcn\" (UniqueName: \"kubernetes.io/projected/0b2553bd-60de-4105-b7c3-a393f1e4cd51-kube-api-access-7rgcn\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:40.613249 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.613208    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:40.613615 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.613596    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b2553bd-60de-4105-b7c3-a393f1e4cd51" containerName="test-curl-7461-140507304161584"
Jul 23 16:35:40.613706 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.613617    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b2553bd-60de-4105-b7c3-a393f1e4cd51" containerName="test-curl-7461-140507304161584"
Jul 23 16:35:40.613778 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.613724    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b2553bd-60de-4105-b7c3-a393f1e4cd51" containerName="test-curl-7461-140507304161584"
Jul 23 16:35:40.616881 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.616860    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:40.624802 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.624775    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:40.686910 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.686875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9gl\" (UniqueName: \"kubernetes.io/projected/25453edc-9e5d-4233-8c8d-0a2254414717-kube-api-access-pb9gl\") pod \"test-curl-7461-140507304293696\" (UID: \"25453edc-9e5d-4233-8c8d-0a2254414717\") " pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:40.787618 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.787583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9gl\" (UniqueName: \"kubernetes.io/projected/25453edc-9e5d-4233-8c8d-0a2254414717-kube-api-access-pb9gl\") pod \"test-curl-7461-140507304293696\" (UID: \"25453edc-9e5d-4233-8c8d-0a2254414717\") " pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:40.799455 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.799419    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9gl\" (UniqueName: \"kubernetes.io/projected/25453edc-9e5d-4233-8c8d-0a2254414717-kube-api-access-pb9gl\") pod \"test-curl-7461-140507304293696\" (UID: \"25453edc-9e5d-4233-8c8d-0a2254414717\") " pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:40.810485 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.810455    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b2553bd-60de-4105-b7c3-a393f1e4cd51" path="/var/lib/kubelet/pods/0b2553bd-60de-4105-b7c3-a393f1e4cd51/volumes"
Jul 23 16:35:40.925973 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:40.925931    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:41.068536 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:41.068313    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:41.070607 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:35:41.070584    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25453edc_9e5d_4233_8c8d_0a2254414717.slice/crio-3c8e7809fd1729bfa4265264f14df14d230607cd78dddf656cf4f5171bc2f71a WatchSource:0}: Error finding container 3c8e7809fd1729bfa4265264f14df14d230607cd78dddf656cf4f5171bc2f71a: Status 404 returned error can't find the container with id 3c8e7809fd1729bfa4265264f14df14d230607cd78dddf656cf4f5171bc2f71a
Jul 23 16:35:41.247864 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:41.247774    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304161584"
Jul 23 16:35:41.247864 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:41.247807    2555 scope.go:117] "RemoveContainer" containerID="c71faa822c49a075dea3a031fa99cbfefb0948fb321f63aaeb075af6ac194de4"
Jul 23 16:35:41.249243 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:41.249001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304293696" event={"ID":"25453edc-9e5d-4233-8c8d-0a2254414717","Type":"ContainerStarted","Data":"3c8e7809fd1729bfa4265264f14df14d230607cd78dddf656cf4f5171bc2f71a"}
Jul 23 16:35:42.255892 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:42.255853    2555 generic.go:356] "Generic (PLEG): container finished" podID="25453edc-9e5d-4233-8c8d-0a2254414717" containerID="1a06316ef8f9463b8ea7e5753f0f4e780464d6f00945efc72efd5e17a6e8a0d1" exitCode=0
Jul 23 16:35:42.256401 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:42.255945    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304293696" event={"ID":"25453edc-9e5d-4233-8c8d-0a2254414717","Type":"ContainerDied","Data":"1a06316ef8f9463b8ea7e5753f0f4e780464d6f00945efc72efd5e17a6e8a0d1"}
Jul 23 16:35:43.391915 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.391889    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:43.413483 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.413454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7461-140507304293696_25453edc-9e5d-4233-8c8d-0a2254414717/test-curl-7461-140507304293696/0.log"
Jul 23 16:35:43.437892 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.437859    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:43.441358 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.441328    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:43.513417 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.513332    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb9gl\" (UniqueName: \"kubernetes.io/projected/25453edc-9e5d-4233-8c8d-0a2254414717-kube-api-access-pb9gl\") pod \"25453edc-9e5d-4233-8c8d-0a2254414717\" (UID: \"25453edc-9e5d-4233-8c8d-0a2254414717\") "
Jul 23 16:35:43.515397 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.515377    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25453edc-9e5d-4233-8c8d-0a2254414717-kube-api-access-pb9gl" (OuterVolumeSpecName: "kube-api-access-pb9gl") pod "25453edc-9e5d-4233-8c8d-0a2254414717" (UID: "25453edc-9e5d-4233-8c8d-0a2254414717"). InnerVolumeSpecName "kube-api-access-pb9gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:35:43.569688 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.569650    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:43.570023 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.570009    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25453edc-9e5d-4233-8c8d-0a2254414717" containerName="test-curl-7461-140507304293696"
Jul 23 16:35:43.570073 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.570024    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="25453edc-9e5d-4233-8c8d-0a2254414717" containerName="test-curl-7461-140507304293696"
Jul 23 16:35:43.570109 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.570103    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="25453edc-9e5d-4233-8c8d-0a2254414717" containerName="test-curl-7461-140507304293696"
Jul 23 16:35:43.572958 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.572939    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:43.577267 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.577241    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:43.614194 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.614161    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb9gl\" (UniqueName: \"kubernetes.io/projected/25453edc-9e5d-4233-8c8d-0a2254414717-kube-api-access-pb9gl\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:43.714913 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.714872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8gc\" (UniqueName: \"kubernetes.io/projected/fe378df0-bc7a-41ea-8492-cc34a1540065-kube-api-access-2h8gc\") pod \"test-curl-7461-140507304293696\" (UID: \"fe378df0-bc7a-41ea-8492-cc34a1540065\") " pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:43.815695 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.815619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8gc\" (UniqueName: \"kubernetes.io/projected/fe378df0-bc7a-41ea-8492-cc34a1540065-kube-api-access-2h8gc\") pod \"test-curl-7461-140507304293696\" (UID: \"fe378df0-bc7a-41ea-8492-cc34a1540065\") " pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:43.824513 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.824490    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8gc\" (UniqueName: \"kubernetes.io/projected/fe378df0-bc7a-41ea-8492-cc34a1540065-kube-api-access-2h8gc\") pod \"test-curl-7461-140507304293696\" (UID: \"fe378df0-bc7a-41ea-8492-cc34a1540065\") " pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:43.884129 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:43.884086    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:44.014064 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:44.014037    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:44.015443 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:35:44.015413    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe378df0_bc7a_41ea_8492_cc34a1540065.slice/crio-a79e317e6a7ad465e6de72c096b56e8d869395b6d3913c435823a8648ed50484 WatchSource:0}: Error finding container a79e317e6a7ad465e6de72c096b56e8d869395b6d3913c435823a8648ed50484: Status 404 returned error can't find the container with id a79e317e6a7ad465e6de72c096b56e8d869395b6d3913c435823a8648ed50484
Jul 23 16:35:44.267142 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:44.267110    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8e7809fd1729bfa4265264f14df14d230607cd78dddf656cf4f5171bc2f71a"
Jul 23 16:35:44.267142 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:44.267136    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:44.268414 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:44.268382    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304293696" event={"ID":"fe378df0-bc7a-41ea-8492-cc34a1540065","Type":"ContainerStarted","Data":"a79e317e6a7ad465e6de72c096b56e8d869395b6d3913c435823a8648ed50484"}
Jul 23 16:35:44.278621 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:44.278593    2555 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7461-140507304293696" oldPodUID="25453edc-9e5d-4233-8c8d-0a2254414717" podUID="fe378df0-bc7a-41ea-8492-cc34a1540065"
Jul 23 16:35:44.809328 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:44.809291    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25453edc-9e5d-4233-8c8d-0a2254414717" path="/var/lib/kubelet/pods/25453edc-9e5d-4233-8c8d-0a2254414717/volumes"
Jul 23 16:35:45.275070 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:45.275030    2555 generic.go:356] "Generic (PLEG): container finished" podID="fe378df0-bc7a-41ea-8492-cc34a1540065" containerID="6157e7cac829c36e790594f68586e8fe31e6971a740a2e2105fd372b55f9b14b" exitCode=0
Jul 23 16:35:45.275256 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:45.275114    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304293696" event={"ID":"fe378df0-bc7a-41ea-8492-cc34a1540065","Type":"ContainerDied","Data":"6157e7cac829c36e790594f68586e8fe31e6971a740a2e2105fd372b55f9b14b"}
Jul 23 16:35:46.416573 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.416546    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:46.437153 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.437124    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7461-140507304293696_fe378df0-bc7a-41ea-8492-cc34a1540065/test-curl-7461-140507304293696/0.log"
Jul 23 16:35:46.464863 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.464830    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:46.468867 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.468839    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7461-140507304293696"]
Jul 23 16:35:46.540087 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.539998    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h8gc\" (UniqueName: \"kubernetes.io/projected/fe378df0-bc7a-41ea-8492-cc34a1540065-kube-api-access-2h8gc\") pod \"fe378df0-bc7a-41ea-8492-cc34a1540065\" (UID: \"fe378df0-bc7a-41ea-8492-cc34a1540065\") "
Jul 23 16:35:46.542149 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.542109    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe378df0-bc7a-41ea-8492-cc34a1540065-kube-api-access-2h8gc" (OuterVolumeSpecName: "kube-api-access-2h8gc") pod "fe378df0-bc7a-41ea-8492-cc34a1540065" (UID: "fe378df0-bc7a-41ea-8492-cc34a1540065"). InnerVolumeSpecName "kube-api-access-2h8gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:35:46.598546 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.598507    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:46.598887 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.598873    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe378df0-bc7a-41ea-8492-cc34a1540065" containerName="test-curl-7461-140507304293696"
Jul 23 16:35:46.598939 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.598889    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe378df0-bc7a-41ea-8492-cc34a1540065" containerName="test-curl-7461-140507304293696"
Jul 23 16:35:46.598974 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.598943    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe378df0-bc7a-41ea-8492-cc34a1540065" containerName="test-curl-7461-140507304293696"
Jul 23 16:35:46.601887 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.601870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:46.608131 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.608104    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:46.641288 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.641232    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h8gc\" (UniqueName: \"kubernetes.io/projected/fe378df0-bc7a-41ea-8492-cc34a1540065-kube-api-access-2h8gc\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:46.742422 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.742381    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9454f\" (UniqueName: \"kubernetes.io/projected/3804866b-26c0-45cc-89fc-ac1b51f5a120-kube-api-access-9454f\") pod \"test-curl-7461-140507304294704\" (UID: \"3804866b-26c0-45cc-89fc-ac1b51f5a120\") " pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:46.810510 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.810424    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe378df0-bc7a-41ea-8492-cc34a1540065" path="/var/lib/kubelet/pods/fe378df0-bc7a-41ea-8492-cc34a1540065/volumes"
Jul 23 16:35:46.843099 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.843061    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9454f\" (UniqueName: \"kubernetes.io/projected/3804866b-26c0-45cc-89fc-ac1b51f5a120-kube-api-access-9454f\") pod \"test-curl-7461-140507304294704\" (UID: \"3804866b-26c0-45cc-89fc-ac1b51f5a120\") " pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:46.851360 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.851337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9454f\" (UniqueName: \"kubernetes.io/projected/3804866b-26c0-45cc-89fc-ac1b51f5a120-kube-api-access-9454f\") pod \"test-curl-7461-140507304294704\" (UID: \"3804866b-26c0-45cc-89fc-ac1b51f5a120\") " pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:46.912336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:46.912292    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:47.048449 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:47.048422    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:47.051218 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:35:47.051187    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3804866b_26c0_45cc_89fc_ac1b51f5a120.slice/crio-d4ab5c0c9eacd2c30ee4ed50dbb97d392bb05bc563c8c817251b1271bedcef7e WatchSource:0}: Error finding container d4ab5c0c9eacd2c30ee4ed50dbb97d392bb05bc563c8c817251b1271bedcef7e: Status 404 returned error can't find the container with id d4ab5c0c9eacd2c30ee4ed50dbb97d392bb05bc563c8c817251b1271bedcef7e
Jul 23 16:35:47.284762 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:47.284715    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304294704" event={"ID":"3804866b-26c0-45cc-89fc-ac1b51f5a120","Type":"ContainerStarted","Data":"d4ab5c0c9eacd2c30ee4ed50dbb97d392bb05bc563c8c817251b1271bedcef7e"}
Jul 23 16:35:47.285956 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:47.285932    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304293696"
Jul 23 16:35:47.286114 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:47.285981    2555 scope.go:117] "RemoveContainer" containerID="6157e7cac829c36e790594f68586e8fe31e6971a740a2e2105fd372b55f9b14b"
Jul 23 16:35:48.143573 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.143531    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n" podUID="5f305559-060c-4bfa-8694-0a6d8baad573" containerName="main" containerID="cri-o://0537f160aa0be3514400f36049c4975a3e0844e58de3eb86902c2bc82f202795" gracePeriod=45
Jul 23 16:35:48.294183 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.294144    2555 generic.go:356] "Generic (PLEG): container finished" podID="5f305559-060c-4bfa-8694-0a6d8baad573" containerID="0537f160aa0be3514400f36049c4975a3e0844e58de3eb86902c2bc82f202795" exitCode=0
Jul 23 16:35:48.294396 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.294235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n" event={"ID":"5f305559-060c-4bfa-8694-0a6d8baad573","Type":"ContainerDied","Data":"0537f160aa0be3514400f36049c4975a3e0844e58de3eb86902c2bc82f202795"}
Jul 23 16:35:48.295782 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.295736    2555 generic.go:356] "Generic (PLEG): container finished" podID="3804866b-26c0-45cc-89fc-ac1b51f5a120" containerID="5338a952e05eee809c4c4d3fac42f6f6a862d77660d91733cb3a471e54c96bdd" exitCode=0
Jul 23 16:35:48.295943 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.295795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304294704" event={"ID":"3804866b-26c0-45cc-89fc-ac1b51f5a120","Type":"ContainerDied","Data":"5338a952e05eee809c4c4d3fac42f6f6a862d77660d91733cb3a471e54c96bdd"}
Jul 23 16:35:48.396230 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.396166    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:35:48.558997 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.558967    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-tmp-dir\") pod \"5f305559-060c-4bfa-8694-0a6d8baad573\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") "
Jul 23 16:35:48.559165 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559009    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-home\") pod \"5f305559-060c-4bfa-8694-0a6d8baad573\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") "
Jul 23 16:35:48.559165 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559069    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pglkj\" (UniqueName: \"kubernetes.io/projected/5f305559-060c-4bfa-8694-0a6d8baad573-kube-api-access-pglkj\") pod \"5f305559-060c-4bfa-8694-0a6d8baad573\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") "
Jul 23 16:35:48.559165 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559097    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-model-cache\") pod \"5f305559-060c-4bfa-8694-0a6d8baad573\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") "
Jul 23 16:35:48.559165 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559124    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-dshm\") pod \"5f305559-060c-4bfa-8694-0a6d8baad573\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") "
Jul 23 16:35:48.559336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559194    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f305559-060c-4bfa-8694-0a6d8baad573-tls-certs\") pod \"5f305559-060c-4bfa-8694-0a6d8baad573\" (UID: \"5f305559-060c-4bfa-8694-0a6d8baad573\") "
Jul 23 16:35:48.559336 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559234    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f305559-060c-4bfa-8694-0a6d8baad573" (UID: "5f305559-060c-4bfa-8694-0a6d8baad573"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:35:48.559432 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559347    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-home" (OuterVolumeSpecName: "home") pod "5f305559-060c-4bfa-8694-0a6d8baad573" (UID: "5f305559-060c-4bfa-8694-0a6d8baad573"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:35:48.559475 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559443    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f305559-060c-4bfa-8694-0a6d8baad573" (UID: "5f305559-060c-4bfa-8694-0a6d8baad573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:35:48.559565 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559544    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-home\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:48.559565 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559562    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-model-cache\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:48.559565 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.559572    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-tmp-dir\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:48.561251 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.561228    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-dshm" (OuterVolumeSpecName: "dshm") pod "5f305559-060c-4bfa-8694-0a6d8baad573" (UID: "5f305559-060c-4bfa-8694-0a6d8baad573"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 16:35:48.561339 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.561302    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f305559-060c-4bfa-8694-0a6d8baad573-kube-api-access-pglkj" (OuterVolumeSpecName: "kube-api-access-pglkj") pod "5f305559-060c-4bfa-8694-0a6d8baad573" (UID: "5f305559-060c-4bfa-8694-0a6d8baad573"). InnerVolumeSpecName "kube-api-access-pglkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:35:48.561339 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.561305    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f305559-060c-4bfa-8694-0a6d8baad573-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f305559-060c-4bfa-8694-0a6d8baad573" (UID: "5f305559-060c-4bfa-8694-0a6d8baad573"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 16:35:48.661012 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.660915    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglkj\" (UniqueName: \"kubernetes.io/projected/5f305559-060c-4bfa-8694-0a6d8baad573-kube-api-access-pglkj\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:48.661012 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.660953    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f305559-060c-4bfa-8694-0a6d8baad573-dshm\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:48.661012 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:48.660965    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f305559-060c-4bfa-8694-0a6d8baad573-tls-certs\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:49.303295 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.303264    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"
Jul 23 16:35:49.303295 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.303282    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n" event={"ID":"5f305559-060c-4bfa-8694-0a6d8baad573","Type":"ContainerDied","Data":"f9037614578d25cc4e8d9f0a5f7d30908396cc181a07542b9aed504aa87044ae"}
Jul 23 16:35:49.303820 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.303332    2555 scope.go:117] "RemoveContainer" containerID="0537f160aa0be3514400f36049c4975a3e0844e58de3eb86902c2bc82f202795"
Jul 23 16:35:49.320852 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.320824    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"]
Jul 23 16:35:49.326359 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.326327    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3a6a71/rate-test-model-3a6a71-kserve-86b8f44b47-kkf4n"]
Jul 23 16:35:49.433638 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.433614    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:49.454387 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.454348    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7461-140507304294704_3804866b-26c0-45cc-89fc-ac1b51f5a120/test-curl-7461-140507304294704/0.log"
Jul 23 16:35:49.478543 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.478504    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:49.482350 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.482324    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:49.570051 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.569963    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9454f\" (UniqueName: \"kubernetes.io/projected/3804866b-26c0-45cc-89fc-ac1b51f5a120-kube-api-access-9454f\") pod \"3804866b-26c0-45cc-89fc-ac1b51f5a120\" (UID: \"3804866b-26c0-45cc-89fc-ac1b51f5a120\") "
Jul 23 16:35:49.572139 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.572112    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3804866b-26c0-45cc-89fc-ac1b51f5a120-kube-api-access-9454f" (OuterVolumeSpecName: "kube-api-access-9454f") pod "3804866b-26c0-45cc-89fc-ac1b51f5a120" (UID: "3804866b-26c0-45cc-89fc-ac1b51f5a120"). InnerVolumeSpecName "kube-api-access-9454f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:35:49.612061 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.612029    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:49.612369 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.612357    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f305559-060c-4bfa-8694-0a6d8baad573" containerName="main"
Jul 23 16:35:49.612413 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.612372    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f305559-060c-4bfa-8694-0a6d8baad573" containerName="main"
Jul 23 16:35:49.612413 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.612386    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3804866b-26c0-45cc-89fc-ac1b51f5a120" containerName="test-curl-7461-140507304294704"
Jul 23 16:35:49.612413 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.612394    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3804866b-26c0-45cc-89fc-ac1b51f5a120" containerName="test-curl-7461-140507304294704"
Jul 23 16:35:49.612506 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.612454    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3804866b-26c0-45cc-89fc-ac1b51f5a120" containerName="test-curl-7461-140507304294704"
Jul 23 16:35:49.612506 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.612463    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f305559-060c-4bfa-8694-0a6d8baad573" containerName="main"
Jul 23 16:35:49.615405 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.615390    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:49.622587 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.622384    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:49.671514 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.671479    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9454f\" (UniqueName: \"kubernetes.io/projected/3804866b-26c0-45cc-89fc-ac1b51f5a120-kube-api-access-9454f\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:49.771994 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.771954    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws45p\" (UniqueName: \"kubernetes.io/projected/3d1abc63-a618-48bb-805c-fc43cac8980d-kube-api-access-ws45p\") pod \"test-curl-7461-140507304294704\" (UID: \"3d1abc63-a618-48bb-805c-fc43cac8980d\") " pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:49.873250 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.873155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws45p\" (UniqueName: \"kubernetes.io/projected/3d1abc63-a618-48bb-805c-fc43cac8980d-kube-api-access-ws45p\") pod \"test-curl-7461-140507304294704\" (UID: \"3d1abc63-a618-48bb-805c-fc43cac8980d\") " pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:49.882338 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.882302    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws45p\" (UniqueName: \"kubernetes.io/projected/3d1abc63-a618-48bb-805c-fc43cac8980d-kube-api-access-ws45p\") pod \"test-curl-7461-140507304294704\" (UID: \"3d1abc63-a618-48bb-805c-fc43cac8980d\") " pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:49.925316 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:49.925279    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:50.059127 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:50.059101    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:50.060635 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:35:50.060609    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1abc63_a618_48bb_805c_fc43cac8980d.slice/crio-65bb6c9cccbfa253777acb11dcd7fbfea02771405b79689e1b23113d73d8cb10 WatchSource:0}: Error finding container 65bb6c9cccbfa253777acb11dcd7fbfea02771405b79689e1b23113d73d8cb10: Status 404 returned error can't find the container with id 65bb6c9cccbfa253777acb11dcd7fbfea02771405b79689e1b23113d73d8cb10
Jul 23 16:35:50.309889 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:50.309853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304294704" event={"ID":"3d1abc63-a618-48bb-805c-fc43cac8980d","Type":"ContainerStarted","Data":"65bb6c9cccbfa253777acb11dcd7fbfea02771405b79689e1b23113d73d8cb10"}
Jul 23 16:35:50.311110 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:50.311084    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:50.311110 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:50.311108    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ab5c0c9eacd2c30ee4ed50dbb97d392bb05bc563c8c817251b1271bedcef7e"
Jul 23 16:35:50.323089 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:50.323063    2555 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7461-140507304294704" oldPodUID="3804866b-26c0-45cc-89fc-ac1b51f5a120" podUID="3d1abc63-a618-48bb-805c-fc43cac8980d"
Jul 23 16:35:50.809902 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:50.809868    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3804866b-26c0-45cc-89fc-ac1b51f5a120" path="/var/lib/kubelet/pods/3804866b-26c0-45cc-89fc-ac1b51f5a120/volumes"
Jul 23 16:35:50.810273 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:50.810260    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f305559-060c-4bfa-8694-0a6d8baad573" path="/var/lib/kubelet/pods/5f305559-060c-4bfa-8694-0a6d8baad573/volumes"
Jul 23 16:35:51.320892 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:51.320857    2555 generic.go:356] "Generic (PLEG): container finished" podID="3d1abc63-a618-48bb-805c-fc43cac8980d" containerID="b16c55050cafb5fb13f5898c293565615f2ac9131b6d5c8915bb9c7b1838b325" exitCode=0
Jul 23 16:35:51.321272 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:51.320923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7461-140507304294704" event={"ID":"3d1abc63-a618-48bb-805c-fc43cac8980d","Type":"ContainerDied","Data":"b16c55050cafb5fb13f5898c293565615f2ac9131b6d5c8915bb9c7b1838b325"}
Jul 23 16:35:52.462191 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.462161    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:52.482421 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.482384    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7461-140507304294704_3d1abc63-a618-48bb-805c-fc43cac8980d/test-curl-7461-140507304294704/0.log"
Jul 23 16:35:52.508394 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.508362    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:52.513407 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.513378    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7461-140507304294704"]
Jul 23 16:35:52.599671 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.599579    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws45p\" (UniqueName: \"kubernetes.io/projected/3d1abc63-a618-48bb-805c-fc43cac8980d-kube-api-access-ws45p\") pod \"3d1abc63-a618-48bb-805c-fc43cac8980d\" (UID: \"3d1abc63-a618-48bb-805c-fc43cac8980d\") "
Jul 23 16:35:52.601703 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.601677    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1abc63-a618-48bb-805c-fc43cac8980d-kube-api-access-ws45p" (OuterVolumeSpecName: "kube-api-access-ws45p") pod "3d1abc63-a618-48bb-805c-fc43cac8980d" (UID: "3d1abc63-a618-48bb-805c-fc43cac8980d"). InnerVolumeSpecName "kube-api-access-ws45p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 16:35:52.700493 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.700452    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws45p\" (UniqueName: \"kubernetes.io/projected/3d1abc63-a618-48bb-805c-fc43cac8980d-kube-api-access-ws45p\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:52.702122 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.702100    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"]
Jul 23 16:35:52.702429 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.702417    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1abc63-a618-48bb-805c-fc43cac8980d" containerName="test-curl-7461-140507304294704"
Jul 23 16:35:52.702466 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.702431    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1abc63-a618-48bb-805c-fc43cac8980d" containerName="test-curl-7461-140507304294704"
Jul 23 16:35:52.702509 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.702500    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1abc63-a618-48bb-805c-fc43cac8980d" containerName="test-curl-7461-140507304294704"
Jul 23 16:35:52.705365 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.705349    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"
Jul 23 16:35:52.707671 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.707651    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6jwk\""
Jul 23 16:35:52.707820 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.707701    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 16:35:52.713031 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.712655    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"]
Jul 23 16:35:52.801654 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.801616    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090b9d4-fd9b-433f-a40a-412620d11847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-759481-xc5wj\" (UID: \"9090b9d4-fd9b-433f-a40a-412620d11847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"
Jul 23 16:35:52.809113 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.809088    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1abc63-a618-48bb-805c-fc43cac8980d" path="/var/lib/kubelet/pods/3d1abc63-a618-48bb-805c-fc43cac8980d/volumes"
Jul 23 16:35:52.902638 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.902598    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090b9d4-fd9b-433f-a40a-412620d11847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-759481-xc5wj\" (UID: \"9090b9d4-fd9b-433f-a40a-412620d11847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"
Jul 23 16:35:52.903257 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:52.903237    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090b9d4-fd9b-433f-a40a-412620d11847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-759481-xc5wj\" (UID: \"9090b9d4-fd9b-433f-a40a-412620d11847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"
Jul 23 16:35:53.017372 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:53.017338    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"
Jul 23 16:35:53.148421 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:53.148395    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"]
Jul 23 16:35:53.150554 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:35:53.150527    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9090b9d4_fd9b_433f_a40a_412620d11847.slice/crio-f5dace5e7933d71b6488ff26a53a68ff4881125d2495febdd3149a2cf3b6d202 WatchSource:0}: Error finding container f5dace5e7933d71b6488ff26a53a68ff4881125d2495febdd3149a2cf3b6d202: Status 404 returned error can't find the container with id f5dace5e7933d71b6488ff26a53a68ff4881125d2495febdd3149a2cf3b6d202
Jul 23 16:35:53.330794 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:53.330723    2555 scope.go:117] "RemoveContainer" containerID="b16c55050cafb5fb13f5898c293565615f2ac9131b6d5c8915bb9c7b1838b325"
Jul 23 16:35:53.331008 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:53.330742    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7461-140507304294704"
Jul 23 16:35:53.332067 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:53.332035    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj" event={"ID":"9090b9d4-fd9b-433f-a40a-412620d11847","Type":"ContainerStarted","Data":"f5dace5e7933d71b6488ff26a53a68ff4881125d2495febdd3149a2cf3b6d202"}
Jul 23 16:35:54.338467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:54.338435    2555 generic.go:356] "Generic (PLEG): container finished" podID="9090b9d4-fd9b-433f-a40a-412620d11847" containerID="fc10cb5137d157b360902914a295ed67283b927a4991adef148b3c9491c5da1e" exitCode=0
Jul 23 16:35:54.338856 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:54.338527    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj" event={"ID":"9090b9d4-fd9b-433f-a40a-412620d11847","Type":"ContainerDied","Data":"fc10cb5137d157b360902914a295ed67283b927a4991adef148b3c9491c5da1e"}
Jul 23 16:35:55.478706 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:55.478674    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"
Jul 23 16:35:55.627399 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:55.627295    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090b9d4-fd9b-433f-a40a-412620d11847-maas-api-service-ca\") pod \"9090b9d4-fd9b-433f-a40a-412620d11847\" (UID: \"9090b9d4-fd9b-433f-a40a-412620d11847\") "
Jul 23 16:35:55.627675 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:55.627651    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9090b9d4-fd9b-433f-a40a-412620d11847-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9090b9d4-fd9b-433f-a40a-412620d11847" (UID: "9090b9d4-fd9b-433f-a40a-412620d11847"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:35:55.728297 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:55.728261    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9090b9d4-fd9b-433f-a40a-412620d11847-maas-api-service-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:35:56.349011 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:56.348971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj" event={"ID":"9090b9d4-fd9b-433f-a40a-412620d11847","Type":"ContainerDied","Data":"f5dace5e7933d71b6488ff26a53a68ff4881125d2495febdd3149a2cf3b6d202"}
Jul 23 16:35:56.349011 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:56.348993    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"
Jul 23 16:35:56.349223 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:35:56.349023    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5dace5e7933d71b6488ff26a53a68ff4881125d2495febdd3149a2cf3b6d202"
Jul 23 16:36:03.272968 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:03.272920    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"]
Jul 23 16:36:03.275989 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:03.275964    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-759481-29747070-sn26b"]
Jul 23 16:36:04.809963 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:04.809927    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a38d7f-3441-4fd7-8e9a-1f618561aff7" path="/var/lib/kubelet/pods/94a38d7f-3441-4fd7-8e9a-1f618561aff7/volumes"
Jul 23 16:36:06.132718 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:06.132683    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:36:07.806860 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:07.806823    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"]
Jul 23 16:36:07.810964 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:07.810938    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-759481-xc5wj"]
Jul 23 16:36:08.809622 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:08.809583    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9090b9d4-fd9b-433f-a40a-412620d11847" path="/var/lib/kubelet/pods/9090b9d4-fd9b-433f-a40a-412620d11847/volumes"
Jul 23 16:36:14.916816 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:14.916776    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"]
Jul 23 16:36:14.917277 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:14.917105    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9090b9d4-fd9b-433f-a40a-412620d11847" containerName="revoke-keys"
Jul 23 16:36:14.917277 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:14.917117    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9090b9d4-fd9b-433f-a40a-412620d11847" containerName="revoke-keys"
Jul 23 16:36:14.917277 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:14.917170    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9090b9d4-fd9b-433f-a40a-412620d11847" containerName="revoke-keys"
Jul 23 16:36:14.920177 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:14.920158    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"
Jul 23 16:36:14.922549 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:14.922527    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6jwk\""
Jul 23 16:36:14.922702 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:14.922578    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 16:36:14.927307 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:14.927281    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"]
Jul 23 16:36:15.006313 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:15.006269    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d512c-2e54-4827-a266-1b851690844f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng\" (UID: \"8f1d512c-2e54-4827-a266-1b851690844f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"
Jul 23 16:36:15.107063 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:15.107021    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d512c-2e54-4827-a266-1b851690844f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng\" (UID: \"8f1d512c-2e54-4827-a266-1b851690844f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"
Jul 23 16:36:15.107640 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:15.107619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d512c-2e54-4827-a266-1b851690844f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng\" (UID: \"8f1d512c-2e54-4827-a266-1b851690844f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"
Jul 23 16:36:15.232038 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:15.231927    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"
Jul 23 16:36:15.572867 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:15.572844    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"]
Jul 23 16:36:15.575015 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:36:15.574978    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1d512c_2e54_4827_a266_1b851690844f.slice/crio-0b3bc4c4d57f20cf1ab5afa417399a09e57e577bc53f2b2ea8bc93e5be237ea7 WatchSource:0}: Error finding container 0b3bc4c4d57f20cf1ab5afa417399a09e57e577bc53f2b2ea8bc93e5be237ea7: Status 404 returned error can't find the container with id 0b3bc4c4d57f20cf1ab5afa417399a09e57e577bc53f2b2ea8bc93e5be237ea7
Jul 23 16:36:16.449021 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:16.448988    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f1d512c-2e54-4827-a266-1b851690844f" containerID="37b928289f49a43067c0446bea20b4410d4683d7ac7714dde2a8cad72583790f" exitCode=0
Jul 23 16:36:16.449417 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:16.449034    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng" event={"ID":"8f1d512c-2e54-4827-a266-1b851690844f","Type":"ContainerDied","Data":"37b928289f49a43067c0446bea20b4410d4683d7ac7714dde2a8cad72583790f"}
Jul 23 16:36:16.449417 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:16.449074    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng" event={"ID":"8f1d512c-2e54-4827-a266-1b851690844f","Type":"ContainerStarted","Data":"0b3bc4c4d57f20cf1ab5afa417399a09e57e577bc53f2b2ea8bc93e5be237ea7"}
Jul 23 16:36:17.585532 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:17.585506    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"
Jul 23 16:36:17.632426 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:17.632393    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d512c-2e54-4827-a266-1b851690844f-maas-api-service-ca\") pod \"8f1d512c-2e54-4827-a266-1b851690844f\" (UID: \"8f1d512c-2e54-4827-a266-1b851690844f\") "
Jul 23 16:36:17.632813 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:17.632789    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1d512c-2e54-4827-a266-1b851690844f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f1d512c-2e54-4827-a266-1b851690844f" (UID: "8f1d512c-2e54-4827-a266-1b851690844f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 16:36:17.733742 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:17.733639    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1d512c-2e54-4827-a266-1b851690844f-maas-api-service-ca\") on node \"ip-10-0-128-121.ec2.internal\" DevicePath \"\""
Jul 23 16:36:18.460408 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:18.460377    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"
Jul 23 16:36:18.460408 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:18.460393    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng" event={"ID":"8f1d512c-2e54-4827-a266-1b851690844f","Type":"ContainerDied","Data":"0b3bc4c4d57f20cf1ab5afa417399a09e57e577bc53f2b2ea8bc93e5be237ea7"}
Jul 23 16:36:18.460627 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:18.460438    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3bc4c4d57f20cf1ab5afa417399a09e57e577bc53f2b2ea8bc93e5be237ea7"
Jul 23 16:36:25.461883 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:25.461803    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"]
Jul 23 16:36:25.467473 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:25.467446    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3a6a71-29747070-c7ptp"]
Jul 23 16:36:26.809431 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:26.809385    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ce5510-e04b-48dc-8cfe-a01819373401" path="/var/lib/kubelet/pods/88ce5510-e04b-48dc-8cfe-a01819373401/volumes"
Jul 23 16:36:28.111408 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:28.111373    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7tk7h"]
Jul 23 16:36:29.998425 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:29.998386    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"]
Jul 23 16:36:30.001069 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:30.001034    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3a6a71-lrlng"]
Jul 23 16:36:30.809680 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:30.809648    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f1d512c-2e54-4827-a266-1b851690844f" path="/var/lib/kubelet/pods/8f1d512c-2e54-4827-a266-1b851690844f/volumes"
Jul 23 16:36:33.079018 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:33.078982    2555 scope.go:117] "RemoveContainer" containerID="21dad5c31c93194e13a8a093b66c74c043c8e383f361bcbfce15d689497498b4"
Jul 23 16:36:33.087302 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:33.087280    2555 scope.go:117] "RemoveContainer" containerID="9dc3a4238bc785ccf9582617a3a15c3ab1d389d569397b1d02df406f46ff02f1"
Jul 23 16:36:37.221607 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:37.221570    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bc77ccfd9-vnwg6_2af302f6-1d3b-4f87-ab06-1c86c307b325/authorino/0.log"
Jul 23 16:36:42.976607 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:42.976575    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rr68k_3312d672-b038-4e98-a46e-e1bbbe6f71be/manager/0.log"
Jul 23 16:36:43.111703 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:43.111668    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5jrwb_1689b2c4-e09a-4d76-b7ac-a0543a62eedd/manager/0.log"
Jul 23 16:36:43.392484 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:43.392454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mcdvg_85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c/server/0.log"
Jul 23 16:36:43.538826 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:43.538792    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xkgwl_819a959a-c089-46a6-b0ed-d326f900f1c4/manager/0.log"
Jul 23 16:36:43.821974 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:43.821891    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-646b5d5c79-g8v4n_b00d5738-76c3-4097-ac99-453517b7cc4f/copy-manifests/0.log"
Jul 23 16:36:43.829515 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:43.829484    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-646b5d5c79-g8v4n_b00d5738-76c3-4097-ac99-453517b7cc4f/manager/0.log"
Jul 23 16:36:44.740380 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:44.740349    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747070-mkh6l_b048d558-3dca-486f-b078-e1766ba115b3/cleanup/0.log"
Jul 23 16:36:46.312898 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:46.312864    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bc77ccfd9-vnwg6_2af302f6-1d3b-4f87-ab06-1c86c307b325/authorino/0.log"
Jul 23 16:36:46.441554 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:46.441523    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pcl7c_4b1e999c-a84e-4456-b43b-a17998c8555c/manager/0.log"
Jul 23 16:36:46.824239 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:46.824210    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bj7kv_b325de6f-af2d-438d-baa7-bcacdc58a159/registry-server/0.log"
Jul 23 16:36:47.078960 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:47.078881    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7tk7h_a3b6746f-21c1-42d7-a41f-b8e107a06389/limitador/0.log"
Jul 23 16:36:47.216058 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:47.216025    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-7nndx_d44de4d5-7b88-4968-bef2-92f432ed3a11/manager/0.log"
Jul 23 16:36:47.605628 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:47.605590    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x66t8_50ba0115-92ad-427f-91a6-4108fb2bda5f/istio-proxy/0.log"
Jul 23 16:36:47.988000 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:47.987974    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb5dcd56-zfn4h_b4d21f9f-fc16-4d61-b17f-0c020e4112d6/kube-auth-proxy/0.log"
Jul 23 16:36:48.122497 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:48.122458    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rp4xm_1c022797-c084-4d04-ac20-05fdf9f9d87b/istio-proxy/1.log"
Jul 23 16:36:49.283695 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:49.283663    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2xdkz_9e371a2f-7379-4784-9b61-efc9f58efba5/main/0.log"
Jul 23 16:36:49.526414 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:49.526388    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6mvwv_b275f199-979b-4429-beb3-3d1f7a55e332/main/0.log"
Jul 23 16:36:58.832635 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:58.832603    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-glxtl_dffd5fde-6708-475a-ae48-e3f9b983009a/global-pull-secret-syncer/0.log"
Jul 23 16:36:58.996488 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:58.996454    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-skg42_fc2c6aa0-ae26-4e3a-883e-9084e0bd1eb9/konnectivity-agent/0.log"
Jul 23 16:36:59.024626 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:36:59.024597    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-121.ec2.internal_d0b8168cb74fd246a62dfc8f2491ac00/haproxy/0.log"
Jul 23 16:37:04.550148 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:04.550112    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bc77ccfd9-vnwg6_2af302f6-1d3b-4f87-ab06-1c86c307b325/authorino/0.log"
Jul 23 16:37:04.614175 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:04.614142    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pcl7c_4b1e999c-a84e-4456-b43b-a17998c8555c/manager/0.log"
Jul 23 16:37:04.727206 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:04.727172    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bj7kv_b325de6f-af2d-438d-baa7-bcacdc58a159/registry-server/0.log"
Jul 23 16:37:04.968497 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:04.968467    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7tk7h_a3b6746f-21c1-42d7-a41f-b8e107a06389/limitador/0.log"
Jul 23 16:37:05.076994 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:05.076957    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-7nndx_d44de4d5-7b88-4968-bef2-92f432ed3a11/manager/0.log"
Jul 23 16:37:07.154267 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:07.154233    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kbp86_404a862c-f038-4943-b5ec-d94dd6cc0282/node-exporter/0.log"
Jul 23 16:37:07.172642 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:07.172614    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kbp86_404a862c-f038-4943-b5ec-d94dd6cc0282/kube-rbac-proxy/0.log"
Jul 23 16:37:07.192467 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:07.192441    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kbp86_404a862c-f038-4943-b5ec-d94dd6cc0282/init-textfile/0.log"
Jul 23 16:37:11.808418 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.808379    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lm22h_0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8/dns/0.log"
Jul 23 16:37:11.815058 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.815028    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"]
Jul 23 16:37:11.815529 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.815507    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f1d512c-2e54-4827-a266-1b851690844f" containerName="revoke-keys"
Jul 23 16:37:11.815529 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.815529    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f1d512c-2e54-4827-a266-1b851690844f" containerName="revoke-keys"
Jul 23 16:37:11.815698 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.815619    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f1d512c-2e54-4827-a266-1b851690844f" containerName="revoke-keys"
Jul 23 16:37:11.818841 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.818822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:11.821507 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.821481    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4nhtn\"/\"kube-root-ca.crt\""
Jul 23 16:37:11.822257 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.822236    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4nhtn\"/\"default-dockercfg-8r4ch\""
Jul 23 16:37:11.822257 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.822251    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4nhtn\"/\"openshift-service-ca.crt\""
Jul 23 16:37:11.829150 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.829125    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"]
Jul 23 16:37:11.849487 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.849461    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lm22h_0f9096f8-4ae2-4bd9-9daa-9fe9499e87a8/kube-rbac-proxy/0.log"
Jul 23 16:37:11.912937 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.912899    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-sys\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:11.913110 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.912963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-podres\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:11.913110 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.913075    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tfw\" (UniqueName: \"kubernetes.io/projected/1800bc85-d1a3-4db8-a0d4-66612a82c12a-kube-api-access-69tfw\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:11.913110 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.913104    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-proc\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:11.913222 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.913139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-lib-modules\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:11.956766 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:11.956723    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gdwzk_8444dfcc-a56d-4220-86eb-2f7b56291263/dns-node-resolver/0.log"
Jul 23 16:37:12.014691 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014652    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tfw\" (UniqueName: \"kubernetes.io/projected/1800bc85-d1a3-4db8-a0d4-66612a82c12a-kube-api-access-69tfw\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.014928 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-proc\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.014928 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014738    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-lib-modules\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.014928 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014809    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-sys\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.014928 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014849    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-podres\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.014928 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014859    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-proc\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.014928 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014922    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-sys\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.015186 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014981    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-podres\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.015186 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.014985    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1800bc85-d1a3-4db8-a0d4-66612a82c12a-lib-modules\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.022820 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.022787    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tfw\" (UniqueName: \"kubernetes.io/projected/1800bc85-d1a3-4db8-a0d4-66612a82c12a-kube-api-access-69tfw\") pod \"perf-node-gather-daemonset-7bshx\" (UID: \"1800bc85-d1a3-4db8-a0d4-66612a82c12a\") " pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.129858 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.129732    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.288565 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.288537    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"]
Jul 23 16:37:12.291112 ip-10-0-128-121 kubenswrapper[2555]: W0723 16:37:12.291082    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1800bc85_d1a3_4db8_a0d4_66612a82c12a.slice/crio-4c0d80bbf4a335cd1a035b05d4fb1e93c40ea08e66381cff73abc8b6713dc3d1 WatchSource:0}: Error finding container 4c0d80bbf4a335cd1a035b05d4fb1e93c40ea08e66381cff73abc8b6713dc3d1: Status 404 returned error can't find the container with id 4c0d80bbf4a335cd1a035b05d4fb1e93c40ea08e66381cff73abc8b6713dc3d1
Jul 23 16:37:12.544345 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.544312    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74d9fffb5-42whm_d9b4b425-1de2-460a-a421-9550a9e3b83c/registry/0.log"
Jul 23 16:37:12.593307 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.593276    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t82bm_1bddfe96-b526-4de7-981b-de1f84b81b3c/node-ca/0.log"
Jul 23 16:37:12.721266 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.721233    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx" event={"ID":"1800bc85-d1a3-4db8-a0d4-66612a82c12a","Type":"ContainerStarted","Data":"4bb69fd23e265bdd086dc4fb6bcbaebdd0b4622c7896add46617b6efe18ae5c8"}
Jul 23 16:37:12.721466 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.721285    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx" event={"ID":"1800bc85-d1a3-4db8-a0d4-66612a82c12a","Type":"ContainerStarted","Data":"4c0d80bbf4a335cd1a035b05d4fb1e93c40ea08e66381cff73abc8b6713dc3d1"}
Jul 23 16:37:12.721466 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.721319    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:12.738996 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:12.738943    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx" podStartSLOduration=1.73892859 podStartE2EDuration="1.73892859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 16:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 16:37:12.738188224 +0000 UTC m=+2920.363722700" watchObservedRunningTime="2026-07-23 16:37:12.73892859 +0000 UTC m=+2920.364463066"
Jul 23 16:37:13.551886 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:13.551854    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x66t8_50ba0115-92ad-427f-91a6-4108fb2bda5f/istio-proxy/0.log"
Jul 23 16:37:13.726552 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:13.726526    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb5dcd56-zfn4h_b4d21f9f-fc16-4d61-b17f-0c020e4112d6/kube-auth-proxy/0.log"
Jul 23 16:37:13.754291 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:13.754261    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rp4xm_1c022797-c084-4d04-ac20-05fdf9f9d87b/istio-proxy/0.log"
Jul 23 16:37:13.806179 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:13.806091    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rp4xm_1c022797-c084-4d04-ac20-05fdf9f9d87b/istio-proxy/1.log"
Jul 23 16:37:14.588823 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:14.588790    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g4zf2_10e4692c-174e-4e57-a7b0-7ec577cea880/serve-healthcheck-canary/0.log"
Jul 23 16:37:15.174965 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:15.174926    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9dh8x_eb77c59d-a9b7-40d0-85f6-c0005b97d735/kube-rbac-proxy/0.log"
Jul 23 16:37:15.196488 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:15.196459    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9dh8x_eb77c59d-a9b7-40d0-85f6-c0005b97d735/exporter/0.log"
Jul 23 16:37:15.229044 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:15.229015    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9dh8x_eb77c59d-a9b7-40d0-85f6-c0005b97d735/extractor/0.log"
Jul 23 16:37:17.429067 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:17.429032    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rr68k_3312d672-b038-4e98-a46e-e1bbbe6f71be/manager/0.log"
Jul 23 16:37:17.756909 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:17.756826    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5jrwb_1689b2c4-e09a-4d76-b7ac-a0543a62eedd/manager/0.log"
Jul 23 16:37:17.920984 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:17.920954    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mcdvg_85e86b8f-e985-4d3f-b194-8f0d3c6e3a2c/server/0.log"
Jul 23 16:37:18.029447 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:18.029359    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xkgwl_819a959a-c089-46a6-b0ed-d326f900f1c4/manager/0.log"
Jul 23 16:37:18.106036 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:18.106006    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-646b5d5c79-g8v4n_b00d5738-76c3-4097-ac99-453517b7cc4f/manager/0.log"
Jul 23 16:37:18.124386 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:18.124358    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-646b5d5c79-g8v4n_b00d5738-76c3-4097-ac99-453517b7cc4f/copy-manifests/0.log"
Jul 23 16:37:18.736307 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:18.736272    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4nhtn/perf-node-gather-daemonset-7bshx"
Jul 23 16:37:19.311890 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:19.311859    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x66t8_50ba0115-92ad-427f-91a6-4108fb2bda5f/istio-proxy/0.log"
Jul 23 16:37:19.459446 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:19.459405    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb5dcd56-zfn4h_b4d21f9f-fc16-4d61-b17f-0c020e4112d6/kube-auth-proxy/0.log"
Jul 23 16:37:19.483211 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:19.483176    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rp4xm_1c022797-c084-4d04-ac20-05fdf9f9d87b/istio-proxy/0.log"
Jul 23 16:37:19.532254 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:19.532221    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rp4xm_1c022797-c084-4d04-ac20-05fdf9f9d87b/istio-proxy/1.log"
Jul 23 16:37:20.897474 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:20.897450    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x66t8_50ba0115-92ad-427f-91a6-4108fb2bda5f/istio-proxy/0.log"
Jul 23 16:37:21.059418 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:21.059392    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fb5dcd56-zfn4h_b4d21f9f-fc16-4d61-b17f-0c020e4112d6/kube-auth-proxy/0.log"
Jul 23 16:37:21.077650 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:21.077623    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rp4xm_1c022797-c084-4d04-ac20-05fdf9f9d87b/istio-proxy/0.log"
Jul 23 16:37:21.130437 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:21.130406    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rp4xm_1c022797-c084-4d04-ac20-05fdf9f9d87b/istio-proxy/1.log"
Jul 23 16:37:26.343616 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.343588    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz59x_2dae43a1-ca00-4cf2-b8c0-ba84573a868a/kube-multus-additional-cni-plugins/0.log"
Jul 23 16:37:26.458398 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.458365    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz59x_2dae43a1-ca00-4cf2-b8c0-ba84573a868a/egress-router-binary-copy/0.log"
Jul 23 16:37:26.476163 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.476134    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz59x_2dae43a1-ca00-4cf2-b8c0-ba84573a868a/cni-plugins/0.log"
Jul 23 16:37:26.498594 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.498571    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz59x_2dae43a1-ca00-4cf2-b8c0-ba84573a868a/bond-cni-plugin/0.log"
Jul 23 16:37:26.522569 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.522545    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz59x_2dae43a1-ca00-4cf2-b8c0-ba84573a868a/routeoverride-cni/0.log"
Jul 23 16:37:26.544501 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.544479    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz59x_2dae43a1-ca00-4cf2-b8c0-ba84573a868a/whereabouts-cni-bincopy/0.log"
Jul 23 16:37:26.569014 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.568995    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz59x_2dae43a1-ca00-4cf2-b8c0-ba84573a868a/whereabouts-cni/0.log"
Jul 23 16:37:26.810105 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.810040    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sq4qw_51b14893-b696-45a2-8d6e-78cf129dd263/kube-multus/0.log"
Jul 23 16:37:26.920814 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.920782    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rghs4_46c729a0-b515-4581-9a67-1ee16d3649ab/network-metrics-daemon/0.log"
Jul 23 16:37:26.937198 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:26.937174    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rghs4_46c729a0-b515-4581-9a67-1ee16d3649ab/kube-rbac-proxy/0.log"
Jul 23 16:37:27.773186 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.773158    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-controller/0.log"
Jul 23 16:37:27.792038 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.792004    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/0.log"
Jul 23 16:37:27.792191 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.792052    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovn-acl-logging/1.log"
Jul 23 16:37:27.809737 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.809712    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/kube-rbac-proxy-node/0.log"
Jul 23 16:37:27.831446 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.831425    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/kube-rbac-proxy-ovn-metrics/0.log"
Jul 23 16:37:27.855262 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.855233    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/northd/0.log"
Jul 23 16:37:27.879532 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.879511    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/nbdb/0.log"
Jul 23 16:37:27.905281 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.905259    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/sbdb/0.log"
Jul 23 16:37:27.995200 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:27.995167    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-29d6g_e01e18b9-f07e-494f-9a2d-50f2f70471de/ovnkube-controller/0.log"
Jul 23 16:37:29.662589 ip-10-0-128-121 kubenswrapper[2555]: I0723 16:37:29.662562    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zr2d7_d443f735-cecc-4b70-95e8-3bce5625a11c/network-check-target-container/0.log"