Jul 28 14:22:35.144329 ip-10-0-141-160 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 14:22:35.570220 ip-10-0-141-160 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 14:22:35.570220 ip-10-0-141-160 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 14:22:35.570220 ip-10-0-141-160 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 14:22:35.570220 ip-10-0-141-160 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 14:22:35.570220 ip-10-0-141-160 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 14:22:35.571221 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.571157    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 14:22:35.578307 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578288    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:35.578307 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578305    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:35.578307 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578309    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:35.578307 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578312    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578315    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578318    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578322    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578324    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578328    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578331    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578333    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578336    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578339    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578342    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578344    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578347    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578349    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578352    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578354    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578357    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578359    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578362    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:35.578465 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578364    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578367    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578369    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578372    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578374    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578376    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578379    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578382    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578384    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578387    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578389    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578392    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578395    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578397    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578400    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578403    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578407    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578411    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578414    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:35.579007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578417    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578420    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578422    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578425    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578429    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578431    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578434    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578438    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578441    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578444    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578447    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578449    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578452    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578454    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578457    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578460    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578463    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578465    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578468    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578471    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:35.579516 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578473    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578477    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578479    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578482    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578484    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578487    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578490    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578493    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578496    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578499    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578502    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578504    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578506    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578509    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578511    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578514    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578516    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578519    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578521    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578524    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:35.580021 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578526    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578529    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578532    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578534    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578537    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578539    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578542    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578545    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578548    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578550    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578554    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578558    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578561    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578563    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578566    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578569    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578572    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.578574    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579256    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:35.580562 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579263    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579267    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579270    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579273    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579276    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579278    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579281    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579284    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579286    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579289    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579291    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579294    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579296    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579299    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579302    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579305    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579308    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579311    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579313    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579316    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:35.581045 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579318    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579321    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579324    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579327    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579329    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579332    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579334    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579337    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579340    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579342    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579345    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579348    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579350    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579353    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579358    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579362    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579365    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579367    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579370    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579372    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:35.581560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579375    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579378    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579380    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579383    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579385    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579388    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579390    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579392    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579397    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579400    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579403    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579406    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579409    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579412    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579416    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579418    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579421    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579424    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579426    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:35.582055 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579429    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579432    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579435    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579438    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579440    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579442    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579446    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579448    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579451    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579453    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579456    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579458    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579461    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579463    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579466    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579469    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579471    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579473    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579476    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579478    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:35.582531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579481    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579483    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579486    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579489    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579491    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579494    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579496    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579500    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579503    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579506    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579509    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579512    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579514    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579517    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579520    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579523    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579526    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579529    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579531    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:35.583022 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579864    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579868    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579871    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579874    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579877    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579893    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579897    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579900    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579904    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579906    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579910    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579914    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579917    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579920    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579922    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579925    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579929    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579932    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579934    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:35.583537 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579937    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579940    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579943    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579945    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579948    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579951    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579953    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579957    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579960    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579963    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579966    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579969    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579972    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579974    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579977    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579980    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579983    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579985    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579988    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:35.584029 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579990    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579993    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579995    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.579998    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580000    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580002    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580005    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580008    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580010    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580013    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580015    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580018    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580020    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580023    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580026    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580029    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580045    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580049    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580051    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:35.584525 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580054    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580056    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580059    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580062    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580065    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580067    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580070    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580073    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580077    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580080    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580082    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580085    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580088    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580090    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580093    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580095    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580098    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580100    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580103    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:35.584992 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580105    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580108    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580110    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580113    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580115    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580118    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580121    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580123    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580128    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580130    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580133    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580135    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580139    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580142    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580144    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580147    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580150    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580152    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580155    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580158    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580160    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:35.585526 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580163    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580166    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580225    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580231    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580236    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580240    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580244    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580247    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580251    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580256    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580261    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580266    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580271    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580276    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580281    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580285    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580288    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580290    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580294    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580299    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580303    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580306    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580309    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580312    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 14:22:35.586058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580316    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580319    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580322    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580326    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580329    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580332    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580335    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580338    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580341    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580344    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580348    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580351    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580354    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580357    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580360    2579 flags.go:64] FLAG: --enable-server="true"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580363    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580367    2579 flags.go:64] FLAG: --event-burst="100"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580370    2579 flags.go:64] FLAG: --event-qps="50"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580373    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580376    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580379    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580383    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580386    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580389    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580392    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 28 14:22:35.586702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580395    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580398    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580403    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580406    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580409    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580412    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580415    2579 flags.go:64] FLAG: --feature-gates=""
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580418    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580421    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580424    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580427    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580430    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580433    2579 flags.go:64] FLAG: --help="false"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580436    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-160.ec2.internal"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580439    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580442    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580445    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580449    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580452    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580455    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580458    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580460    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580463    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 14:22:35.587323 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580466    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580469    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580472    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580475    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580478    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580480    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580483    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580486    2579 flags.go:64] FLAG: --lock-file=""
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580489    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580492    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580495    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580501    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580504    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580507    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580510    2579 flags.go:64] FLAG: --logging-format="text"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580513    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580516    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580519    2579 flags.go:64] FLAG: --manifest-url=""
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580522    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580528    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580531    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580535    2579 flags.go:64] FLAG: --max-pods="110"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580537    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580540    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580543    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 14:22:35.587919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580546    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580549    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580552    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580555    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580562    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580565    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580568    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580572    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580575    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580581    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580584    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580587    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580590    2579 flags.go:64] FLAG: --port="10250"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580593    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580596    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ea3e799f2b06e580"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580599    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580602    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580605    2579 flags.go:64] FLAG: --register-node="true"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580607    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580612    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580615    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580618    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580621    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580629    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580632    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 14:22:35.588547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580635    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580639    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580643    2579 flags.go:64] FLAG: --runonce="false"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580645    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580648    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580651    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580654    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580657    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580660    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580663    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580666    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580669    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580672    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580675    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580677    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580680    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580683    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580686    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580691    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580694    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580697    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580701    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580703    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580706    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580709    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 14:22:35.589187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580712    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580715    2579 flags.go:64] FLAG: --v="2"
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580719    2579 flags.go:64] FLAG: --version="false"
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580723    2579 flags.go:64] FLAG: --vmodule=""
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580727    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.580730    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580833    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580837    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580842    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580845    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580848    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580850    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580853    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580856    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580858    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580861    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580864    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580866    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580869    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580872    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580874    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:35.589823 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580877    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580880    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580882    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580885    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580888    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580890    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580893    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580896    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580899    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580901    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580904    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580906    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580909    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580912    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580914    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580918    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580921    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580924    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580926    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580930    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:35.590462 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580932    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580935    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580937    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580940    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580942    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580946    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580949    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580952    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580955    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580958    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580960    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580962    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580965    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580968    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580971    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580973    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580976    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580978    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580981    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:35.590958 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580983    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580986    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580988    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580991    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580993    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580996    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.580999    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581002    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581004    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581007    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581010    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581012    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581016    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581019    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581021    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581025    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581029    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581045    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581048    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:35.591442 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581051    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581054    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581057    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581059    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581062    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581065    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581067    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581070    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581072    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581075    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581079    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581082    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581085    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581088    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581091    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581093    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581096    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581099    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581103    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581106    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:35.591914 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581108    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581111    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581113    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581117    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581120    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.581124    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.581132    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.587587    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.587599    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.588560    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.588568    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588623    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588628    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588632    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:35.592409 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588635    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588638    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588641    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588644    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588646    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588649    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588651    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588654    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588658    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588662    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588665    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588669    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588671    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588674    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588678    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588682    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588685    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588688    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588690    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:35.592759 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588693    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588696    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588699    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588701    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588704    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588706    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588709    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588712    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588715    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588718    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588720    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588723    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588726    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588728    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588731    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588734    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588736    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588739    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588741    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588744    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:35.593250 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588746    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588749    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588751    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588754    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588756    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588759    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588761    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588764    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588766    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588769    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588772    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588774    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588777    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588779    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588782    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588785    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588787    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588790    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588793    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588797    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:35.593726 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588799    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588802    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588804    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588807    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588810    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588812    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588815    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588817    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588820    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588822    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588824    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588827    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588830    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588832    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588835    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588837    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588841    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588844    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588847    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588850    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:35.594221 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588852    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588855    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588858    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588860    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588863    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588866    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588868    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588871    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588874    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588877    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588879    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588882    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588885    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588887    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588895    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588898    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:35.594716 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.588901    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.588906    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589007    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589012    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589015    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589018    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589020    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589023    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589026    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589028    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589048    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589052    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589055    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589059    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 14:22:35.595188 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589063    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589066    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589069    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589072    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589075    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589078    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589081    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589084    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589088    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589091    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589093    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589097    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589100    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589103    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589106    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589109    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589112    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589115    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589117    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589120    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 14:22:35.595531 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589123    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589125    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589128    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589130    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589132    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589135    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589137    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589140    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589142    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589145    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589147    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589150    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589152    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589155    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589157    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589160    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589163    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589165    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589167    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589171    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 14:22:35.596060 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589173    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589176    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589180    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589183    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589187    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589190    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589193    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589195    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589198    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589201    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589204    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589206    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589209    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589212    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589214    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589217    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589219    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589222    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589224    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 14:22:35.596551 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589227    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589229    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589232    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589234    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589237    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589240    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589250    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589253    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589256    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589258    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589261    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589263    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589266    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589268    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589271    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589273    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589276    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589279    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589282    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589285    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 14:22:35.596998 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589287    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589290    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589292    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589294    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589297    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589300    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589302    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:35.589305    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.589310    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.589393    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.591747    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.593481    2579 server.go:1019] "Starting client certificate rotation"
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.593579    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 14:22:35.597511 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.593605    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 14:22:35.597834 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.597193    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 14:22:35.598418 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.598403    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 14:22:35.610482 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.610445    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 28 14:22:35.613691 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.613673    2579 log.go:25] "Validated CRI v1 image API"
Jul 28 14:22:35.614492 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.614480    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 14:22:35.618433 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.618409    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9b3870d6-fd09-4d4c-a2f3-f76e398843e8:/dev/nvme0n1p3 b78dc5f6-e1e6-467f-a77d-0a88b45544fc:/dev/nvme0n1p4]
Jul 28 14:22:35.618479 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.618433    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 14:22:35.624599 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.624465    2579 manager.go:216] Machine: {Timestamp:2026-07-28 14:22:35.622138388 +0000 UTC m=+0.351344959 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104916 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dfd7404f421432c67ae77016a0494 SystemUUID:ec2dfd74-04f4-2143-2c67-ae77016a0494 BootID:4cb81276-11c6-4069-800a-37a6db77031c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9c:66:01:5c:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9c:66:01:5c:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:b4:0c:37:96:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 14:22:35.624599 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.624589    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 14:22:35.624730 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.624655    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 14:22:35.627295 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.627280    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 14:22:35.628428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.628401    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 14:22:35.628548 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.628432    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 14:22:35.629861 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.629851    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 14:22:35.629898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.629862    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 14:22:35.629898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.629870    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 14:22:35.630708 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.630698    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 14:22:35.631898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.631888    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 14:22:35.633695 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.633684    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 14:22:35.633796 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.633787    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 14:22:35.633825 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.633816    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 14:22:35.633852 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.633825    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 14:22:35.633852 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.633839    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 14:22:35.633852 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.633847    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 28 14:22:35.633958 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.633854    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 14:22:35.634179 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.634163    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 14:22:35.634240 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.634186    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 14:22:35.636577 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.636560    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 14:22:35.639181 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.639167    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 14:22:35.639237 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.639197    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 14:22:35.641275 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641261    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 14:22:35.641317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641282    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 14:22:35.641317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641292    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 14:22:35.641317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641300    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 14:22:35.641317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641313    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 14:22:35.641428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641322    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 14:22:35.641428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641331    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 14:22:35.641428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641336    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 14:22:35.641428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641342    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 14:22:35.641428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641347    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 14:22:35.641428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641364    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 14:22:35.641428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.641369    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 14:22:35.643101 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.643091    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 14:22:35.643101 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.643101    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 14:22:35.645742 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.645724    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 14:22:35.645742 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.645722    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 14:22:35.645873 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.645798    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 14:22:35.646184 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.646156    2579 server.go:1268] "Started kubelet"
Jul 28 14:22:35.646606 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.646577    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 14:22:35.646686 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.646637    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 14:22:35.646738 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.646581    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 14:22:35.646873 ip-10-0-141-160 systemd[1]: Started Kubernetes Kubelet.
Jul 28 14:22:35.649007 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.648991    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 14:22:35.649991 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.649978    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 14:22:35.655054 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.655015    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 14:22:35.655959 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.654946    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-160.ec2.internal.18c67a1a4b10f61e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-160.ec2.internal,UID:ip-10-0-141-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-160.ec2.internal,},FirstTimestamp:2026-07-28 14:22:35.646121502 +0000 UTC m=+0.375328076,LastTimestamp:2026-07-28 14:22:35.646121502 +0000 UTC m=+0.375328076,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-160.ec2.internal,}"
Jul 28 14:22:35.659339 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.659319    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 14:22:35.659339 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.659329    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 14:22:35.659457 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.659399    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:35.660090 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660073    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 14:22:35.660090 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660074    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 14:22:35.660268 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660098    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 14:22:35.660268 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660185    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 14:22:35.660268 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660192    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 14:22:35.660763 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660747    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 14:22:35.660763 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660761    2579 factory.go:55] Registering systemd factory
Jul 28 14:22:35.660884 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660767    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 28 14:22:35.660992 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660973    2579 factory.go:153] Registering CRI-O factory
Jul 28 14:22:35.661074 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.660995    2579 factory.go:223] Registration of the crio container factory successfully
Jul 28 14:22:35.661074 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.661024    2579 factory.go:103] Registering Raw factory
Jul 28 14:22:35.661074 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.661053    2579 manager.go:1196] Started watching for new ooms in manager
Jul 28 14:22:35.661463 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.661446    2579 manager.go:349] Starting recovery of all containers
Jul 28 14:22:35.671147 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.671127    2579 manager.go:354] Recovery completed
Jul 28 14:22:35.671464 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.671438    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 14:22:35.671539 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.671483    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 14:22:35.676073 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.676058    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:35.677903 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.677882    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:35.677960 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.677915    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:35.677960 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.677927    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:35.678453 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.678440    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:35.679468 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.679453    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 14:22:35.679468 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.679464    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 14:22:35.679588 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.679480    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 14:22:35.680107 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.680090    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:35.680163 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.680120    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:35.680163 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.680136    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:35.681826 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.681811    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrq7b"
Jul 28 14:22:35.688302 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.688227    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-160.ec2.internal.18c67a1a4cf5e9b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-160.ec2.internal,UID:ip-10-0-141-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-160.ec2.internal,},FirstTimestamp:2026-07-28 14:22:35.677903281 +0000 UTC m=+0.407109851,LastTimestamp:2026-07-28 14:22:35.677903281 +0000 UTC m=+0.407109851,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-160.ec2.internal,}"
Jul 28 14:22:35.690554 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.690539    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrq7b"
Jul 28 14:22:35.699797 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.699732    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-160.ec2.internal.18c67a1a4cf62887  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-160.ec2.internal,UID:ip-10-0-141-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-160.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-160.ec2.internal,},FirstTimestamp:2026-07-28 14:22:35.677919367 +0000 UTC m=+0.407125938,LastTimestamp:2026-07-28 14:22:35.677919367 +0000 UTC m=+0.407125938,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-160.ec2.internal,}"
Jul 28 14:22:35.711851 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.711784    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-160.ec2.internal.18c67a1a4cf65d10  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-160.ec2.internal,UID:ip-10-0-141-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-160.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-160.ec2.internal,},FirstTimestamp:2026-07-28 14:22:35.677932816 +0000 UTC m=+0.407139388,LastTimestamp:2026-07-28 14:22:35.677932816 +0000 UTC m=+0.407139388,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-160.ec2.internal,}"
Jul 28 14:22:35.724072 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.723988    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-160.ec2.internal.18c67a1a4cf5e9b1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-160.ec2.internal.18c67a1a4cf5e9b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-160.ec2.internal,UID:ip-10-0-141-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-160.ec2.internal,},FirstTimestamp:2026-07-28 14:22:35.677903281 +0000 UTC m=+0.407109851,LastTimestamp:2026-07-28 14:22:35.680106916 +0000 UTC m=+0.409313492,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-160.ec2.internal,}"
Jul 28 14:22:35.733099 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.733020    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-160.ec2.internal.18c67a1a4cf62887\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-160.ec2.internal.18c67a1a4cf62887  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-160.ec2.internal,UID:ip-10-0-141-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-160.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-160.ec2.internal,},FirstTimestamp:2026-07-28 14:22:35.677919367 +0000 UTC m=+0.407125938,LastTimestamp:2026-07-28 14:22:35.680128083 +0000 UTC m=+0.409334658,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-160.ec2.internal,}"
Jul 28 14:22:35.743498 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.743439    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-160.ec2.internal.18c67a1a4cf65d10\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-160.ec2.internal.18c67a1a4cf65d10  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-160.ec2.internal,UID:ip-10-0-141-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-160.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-160.ec2.internal,},FirstTimestamp:2026-07-28 14:22:35.677932816 +0000 UTC m=+0.407139388,LastTimestamp:2026-07-28 14:22:35.680141402 +0000 UTC m=+0.409347973,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-160.ec2.internal,}"
Jul 28 14:22:35.759607 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.759585    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:35.830771 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.830720    2579 policy_none.go:49] "None policy: Start"
Jul 28 14:22:35.830771 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.830752    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 14:22:35.830771 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.830764    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 14:22:35.831704 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.831694    2579 policy_none.go:47] "Start"
Jul 28 14:22:35.858557 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.858545    2579 manager.go:337] "Starting Device Plugin manager"
Jul 28 14:22:35.858631 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.858571    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 14:22:35.858631 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.858581    2579 server.go:87] "Starting device plugin registration server"
Jul 28 14:22:35.858769 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.858756    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 14:22:35.858810 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.858769    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 14:22:35.858901 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.858883    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 14:22:35.858991 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.858975    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 14:22:35.859115 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.859003    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 14:22:35.859498 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.859476    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 14:22:35.859561 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.859523    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:35.884136 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.884119    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-160.ec2.internal\" not found" node="ip-10-0-141-160.ec2.internal"
Jul 28 14:22:35.959701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.959680    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:35.960472 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.960456    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:35.960554 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.960484    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:35.960554 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.960494    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:35.960554 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.960516    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-160.ec2.internal"
Jul 28 14:22:35.961637 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.961613    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 14:22:35.962863 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.962845    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 14:22:35.962942 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.962877    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 14:22:35.962942 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.962895    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 14:22:35.963025 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.962930    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 14:22:35.964841 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.964823    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:35.967179 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:35.967163    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-160.ec2.internal"
Jul 28 14:22:35.967254 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:35.967185    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-160.ec2.internal\": node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:36.018344 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:36.018323    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:36.063680 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.063650    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal"]
Jul 28 14:22:36.063774 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.063737    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:36.066267 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.066253    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:36.066363 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.066284    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:36.066363 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.066298    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:36.067555 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.067541    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:36.067681 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.067666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.067729 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.067696    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:36.069959 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.069943    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:36.070062 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.069946    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:36.070062 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.069972    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:36.070062 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.069985    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:36.070062 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.069985    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:36.070062 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.069999    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:36.071287 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.071272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.071351 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.071297    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 14:22:36.072071 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.072056    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 14:22:36.072152 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.072085    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 14:22:36.072152 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.072097    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 14:22:36.095624 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:36.095582    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-160.ec2.internal\" not found" node="ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.099707 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:36.099692    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-160.ec2.internal\" not found" node="ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.119278 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:36.119260    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:36.163553 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.163537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/195258ab55c23e56ac6696c29a5c10c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal\" (UID: \"195258ab55c23e56ac6696c29a5c10c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.163627 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.163562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/195258ab55c23e56ac6696c29a5c10c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal\" (UID: \"195258ab55c23e56ac6696c29a5c10c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.163627 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.163578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a6857bda9fd953712469ebe65794e79-config\") pod \"kube-apiserver-proxy-ip-10-0-141-160.ec2.internal\" (UID: \"1a6857bda9fd953712469ebe65794e79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.219935 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:36.219917    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:36.263867 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.263848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/195258ab55c23e56ac6696c29a5c10c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal\" (UID: \"195258ab55c23e56ac6696c29a5c10c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.263923 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.263871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/195258ab55c23e56ac6696c29a5c10c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal\" (UID: \"195258ab55c23e56ac6696c29a5c10c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.263923 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.263891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a6857bda9fd953712469ebe65794e79-config\") pod \"kube-apiserver-proxy-ip-10-0-141-160.ec2.internal\" (UID: \"1a6857bda9fd953712469ebe65794e79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.264016 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.263931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a6857bda9fd953712469ebe65794e79-config\") pod \"kube-apiserver-proxy-ip-10-0-141-160.ec2.internal\" (UID: \"1a6857bda9fd953712469ebe65794e79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.264016 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.263961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/195258ab55c23e56ac6696c29a5c10c0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal\" (UID: \"195258ab55c23e56ac6696c29a5c10c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.264016 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.263966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/195258ab55c23e56ac6696c29a5c10c0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal\" (UID: \"195258ab55c23e56ac6696c29a5c10c0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.320949 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:36.320932    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:36.398483 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.398437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.403090 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:36.402965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:36.421802 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:36.421781    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-160.ec2.internal\" not found"
Jul 28 14:22:37.735382 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.735366    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:37.736979 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.736965    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:37.742070 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.742051    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:37.767814 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.767798    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 14:22:37.768921 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.768909    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal"
Jul 28 14:22:37.780895 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.780880    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 14:22:37.874476 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.874446    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 14:22:37.874796 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.874599    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 14:22:37.874796 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.874618    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 14:22:37.916387 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.916369    2579 apiserver.go:52] "Watching apiserver"
Jul 28 14:22:37.921376 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.921361    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 14:22:37.923364 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.923343    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-975st","kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2","openshift-cluster-node-tuning-operator/tuned-298tw","openshift-multus/multus-additional-cni-plugins-sczc4","openshift-multus/multus-l6zvc","openshift-multus/network-metrics-daemon-h5w88","openshift-network-diagnostics/network-check-target-92xq5","kube-system/global-pull-secret-syncer-txplh","openshift-dns/node-resolver-nxbfp","openshift-image-registry/node-ca-wrgjj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal","openshift-network-operator/iptables-alerter-x4kps","openshift-ovn-kubernetes/ovnkube-node-xvvg4"]
Jul 28 14:22:37.924580 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.924561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-975st"
Jul 28 14:22:37.925651 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.925623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:37.926602 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.926587    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.927118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.927101    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 14:22:37.927196 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.927107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 14:22:37.927538 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.927520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rjqjh\""
Jul 28 14:22:37.927637 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.927622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.928683 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.928664    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.928771 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.928755    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tcrx4\""
Jul 28 14:22:37.929085 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.929067    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 14:22:37.929833 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.929804    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 14:22:37.929947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.929833    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 14:22:37.929947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.929878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 14:22:37.929947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.929885    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 14:22:37.930087 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:37.930143 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 14:22:37.930143 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930127    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l5w6j\""
Jul 28 14:22:37.930240 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:37.930137    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:37.930240 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930152    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-chbrv\""
Jul 28 14:22:37.930240 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930214    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 14:22:37.930360 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930256    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 14:22:37.930656 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930640    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 14:22:37.930713 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930670    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 14:22:37.930787 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930770    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 14:22:37.930918 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.930886    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zttf7\""
Jul 28 14:22:37.931185 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.931174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:37.931246 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:37.931230    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:37.932281 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.932266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:37.932367 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:37.932349    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:37.933587 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.933568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:37.934717 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.934704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:37.935893 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.935876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:37.937029 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.937012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.939698 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.939679    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vpwgk\""
Jul 28 14:22:37.941277 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.941258    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 14:22:37.941361 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.941340    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 14:22:37.942311 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942019    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 14:22:37.942311 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942140    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 14:22:37.942311 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942158    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 14:22:37.942311 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942218    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 14:22:37.942311 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942244    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 14:22:37.942625 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942350    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 14:22:37.942625 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 14:22:37.942625 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 14:22:37.942764 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942692    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 14:22:37.943557 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.942979    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cqncs\""
Jul 28 14:22:37.943557 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.943205    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gvrj\""
Jul 28 14:22:37.943557 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.943281    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 14:22:37.944054 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.943786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 14:22:37.944054 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.943934    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 14:22:37.944054 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.944041    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 14:22:37.944824 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.944798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9c2vm\""
Jul 28 14:22:37.945193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.945177    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 14:22:37.948490 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-os-release\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.948566 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-hosts-file\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:37.948566 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntbp\" (UniqueName: \"kubernetes.io/projected/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-kube-api-access-vntbp\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:37.948566 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-systemd\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.948566 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-var-lib-kubelet\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.948566 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-cni-bin\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.948716 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-kubelet\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.948716 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-cni-bin\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.948716 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cnibin\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.948716 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-dbus\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:37.948716 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-kubernetes\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.948716 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysctl-d\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcaae002-b584-4e0a-b423-86454a1a68cc-cni-binary-copy\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-etc-kubernetes\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36bf149a-2851-4d65-ac83-cd613597aae1-host-slash\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-registration-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffdl2\" (UniqueName: \"kubernetes.io/projected/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-kube-api-access-ffdl2\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-hostroot\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-daemon-config\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.948947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-run-netns\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8q7t\" (UniqueName: \"kubernetes.io/projected/bf76e089-703b-4a88-ab06-e23f370322bf-kube-api-access-d8q7t\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.948989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870f6c21-2fc5-436b-9f77-77b53329ca23-konnectivity-ca\") pod \"konnectivity-agent-975st\" (UID: \"870f6c21-2fc5-436b-9f77-77b53329ca23\") " pod="kube-system/konnectivity-agent-975st"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-system-cni-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-multus-certs\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-node-log\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870f6c21-2fc5-436b-9f77-77b53329ca23-agent-certs\") pod \"konnectivity-agent-975st\" (UID: \"870f6c21-2fc5-436b-9f77-77b53329ca23\") " pod="kube-system/konnectivity-agent-975st"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6921424a-8439-4602-acc4-e525c0779616-host\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwvr\" (UniqueName: \"kubernetes.io/projected/dcaae002-b584-4e0a-b423-86454a1a68cc-kube-api-access-jhwvr\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-slash\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-env-overrides\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-kubelet-config\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysconfig\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36bf149a-2851-4d65-ac83-cd613597aae1-iptables-alerter-script\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cni-binary-copy\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-tmp-dir\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-host\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.949325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-tmp\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-kubelet\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxl8m\" (UniqueName: \"kubernetes.io/projected/36bf149a-2851-4d65-ac83-cd613597aae1-kube-api-access-pxl8m\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-var-lib-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-etc-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-run\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-cni-multus\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-netns\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf76e089-703b-4a88-ab06-e23f370322bf-ovn-node-metrics-cert\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-socket-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-device-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8vp\" (UniqueName: \"kubernetes.io/projected/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-kube-api-access-4g8vp\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-cnibin\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-ovnkube-script-lib\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.949973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6921424a-8439-4602-acc4-e525c0779616-serviceca\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4v8q\" (UniqueName: \"kubernetes.io/projected/6921424a-8439-4602-acc4-e525c0779616-kube-api-access-r4v8q\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-systemd-units\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-systemd\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-lib-modules\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-cni-netd\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-cni-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-os-release\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-k8s-cni-cncf-io\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-conf-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.949958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-ovn\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-ovnkube-config\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.950634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-log-socket\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-sys-fs\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-system-cni-dir\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764g5\" (UniqueName: \"kubernetes.io/projected/4ca104f8-d1bd-4ea6-b807-b05829d737fe-kube-api-access-764g5\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysctl-conf\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-tuned\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-socket-dir-parent\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-etc-selinux\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xft7k\" (UniqueName: \"kubernetes.io/projected/c4eb704f-bf54-463d-a108-fe87f61a4e44-kube-api-access-xft7k\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-modprobe-d\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.951193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.950437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-sys\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:37.953763 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.953746    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 14:22:37.988890 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:37.988871    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-766fj"
Jul 28 14:22:38.000724 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.000703    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-766fj"
Jul 28 14:22:38.050670 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-run-netns\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.050749 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8q7t\" (UniqueName: \"kubernetes.io/projected/bf76e089-703b-4a88-ab06-e23f370322bf-kube-api-access-d8q7t\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.050749 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870f6c21-2fc5-436b-9f77-77b53329ca23-konnectivity-ca\") pod \"konnectivity-agent-975st\" (UID: \"870f6c21-2fc5-436b-9f77-77b53329ca23\") " pod="kube-system/konnectivity-agent-975st"
Jul 28 14:22:38.050749 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-system-cni-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.050878 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-multus-certs\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.050878 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-run-netns\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.050878 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-multus-certs\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.050878 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-system-cni-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.051125 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-node-log\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.051125 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870f6c21-2fc5-436b-9f77-77b53329ca23-agent-certs\") pod \"konnectivity-agent-975st\" (UID: \"870f6c21-2fc5-436b-9f77-77b53329ca23\") " pod="kube-system/konnectivity-agent-975st"
Jul 28 14:22:38.051125 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6921424a-8439-4602-acc4-e525c0779616-host\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:38.051125 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.050991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-node-log\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.051125 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6921424a-8439-4602-acc4-e525c0779616-host\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:38.051125 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwvr\" (UniqueName: \"kubernetes.io/projected/dcaae002-b584-4e0a-b423-86454a1a68cc-kube-api-access-jhwvr\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.051125 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-slash\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-env-overrides\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-kubelet-config\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysconfig\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-slash\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36bf149a-2851-4d65-ac83-cd613597aae1-iptables-alerter-script\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysconfig\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-kubelet-config\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cni-binary-copy\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-tmp-dir\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051258    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/870f6c21-2fc5-436b-9f77-77b53329ca23-konnectivity-ca\") pod \"konnectivity-agent-975st\" (UID: \"870f6c21-2fc5-436b-9f77-77b53329ca23\") " pod="kube-system/konnectivity-agent-975st"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-host\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-tmp\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-kubelet\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxl8m\" (UniqueName: \"kubernetes.io/projected/36bf149a-2851-4d65-ac83-cd613597aae1-kube-api-access-pxl8m\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-var-lib-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.051457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-etc-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-run\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-cni-multus\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-tmp-dir\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-netns\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf76e089-703b-4a88-ab06-e23f370322bf-ovn-node-metrics-cert\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-socket-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-var-lib-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-device-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8vp\" (UniqueName: \"kubernetes.io/projected/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-kube-api-access-4g8vp\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-cnibin\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-socket-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-ovnkube-script-lib\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-run\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-env-overrides\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052118 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/36bf149a-2851-4d65-ac83-cd613597aae1-iptables-alerter-script\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.051969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cni-binary-copy\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6921424a-8439-4602-acc4-e525c0779616-serviceca\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-etc-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-cnibin\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-cni-multus\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-netns\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4v8q\" (UniqueName: \"kubernetes.io/projected/6921424a-8439-4602-acc4-e525c0779616-kube-api-access-r4v8q\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.052428    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-systemd-units\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-systemd\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.052491    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret podName:a00f8f97-a454-45d2-9758-e09e9f0ff6f3 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:38.552472469 +0000 UTC m=+1.999321779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret") pod "global-pull-secret-syncer-txplh" (UID: "a00f8f97-a454-45d2-9758-e09e9f0ff6f3") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-systemd\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-lib-modules\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-cni-netd\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.052844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-systemd-units\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-cni-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-device-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-lib-modules\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-openvswitch\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-cni-netd\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-os-release\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-kubelet\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-cni-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-k8s-cni-cncf-io\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-run-k8s-cni-cncf-io\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-conf-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-ovn\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6921424a-8439-4602-acc4-e525c0779616-serviceca\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-run-ovn\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-ovnkube-config\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.053678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-os-release\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-conf-dir\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-log-socket\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.052319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-host\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-log-socket\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-sys-fs\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-sys-fs\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-system-cni-dir\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-system-cni-dir\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764g5\" (UniqueName: \"kubernetes.io/projected/4ca104f8-d1bd-4ea6-b807-b05829d737fe-kube-api-access-764g5\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-ovnkube-config\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.054501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysctl-conf\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-tuned\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-socket-dir-parent\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-etc-selinux\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xft7k\" (UniqueName: \"kubernetes.io/projected/c4eb704f-bf54-463d-a108-fe87f61a4e44-kube-api-access-xft7k\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-modprobe-d\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-sys\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-os-release\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-hosts-file\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf76e089-703b-4a88-ab06-e23f370322bf-ovnkube-script-lib\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.053988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntbp\" (UniqueName: \"kubernetes.io/projected/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-kube-api-access-vntbp\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-etc-selinux\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-systemd\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-socket-dir-parent\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-var-lib-kubelet\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-cni-bin\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.055231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-kubelet\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-cni-bin\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysctl-conf\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-hosts-file\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-os-release\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-host-var-lib-cni-bin\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-kubelet\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cnibin\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf76e089-703b-4a88-ab06-e23f370322bf-host-cni-bin\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-sys\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ca104f8-d1bd-4ea6-b807-b05829d737fe-cnibin\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-dbus\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-kubernetes\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysctl-d\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcaae002-b584-4e0a-b423-86454a1a68cc-cni-binary-copy\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-modprobe-d\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-kubernetes\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.055898 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-sysctl-d\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-dbus\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-systemd\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.054794    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.054835    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs podName:ed53709b-b3b8-4ef7-a9c2-35a9789dac55 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:38.554820332 +0000 UTC m=+2.001669619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs") pod "network-metrics-daemon-h5w88" (UID: "ed53709b-b3b8-4ef7-a9c2-35a9789dac55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-var-lib-kubelet\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-etc-kubernetes\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36bf149a-2851-4d65-ac83-cd613597aae1-host-slash\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.054985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-registration-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcaae002-b584-4e0a-b423-86454a1a68cc-cni-binary-copy\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-etc-kubernetes\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffdl2\" (UniqueName: \"kubernetes.io/projected/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-kube-api-access-ffdl2\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36bf149a-2851-4d65-ac83-cd613597aae1-host-slash\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-tmp\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4eb704f-bf54-463d-a108-fe87f61a4e44-registration-dir\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-hostroot\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.056374 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-daemon-config\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.056899 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf76e089-703b-4a88-ab06-e23f370322bf-ovn-node-metrics-cert\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.056899 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcaae002-b584-4e0a-b423-86454a1a68cc-hostroot\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.056899 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055507    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ca104f8-d1bd-4ea6-b807-b05829d737fe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.056899 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/870f6c21-2fc5-436b-9f77-77b53329ca23-agent-certs\") pod \"konnectivity-agent-975st\" (UID: \"870f6c21-2fc5-436b-9f77-77b53329ca23\") " pod="kube-system/konnectivity-agent-975st"
Jul 28 14:22:38.056899 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.055827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcaae002-b584-4e0a-b423-86454a1a68cc-multus-daemon-config\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.056899 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.056604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-etc-tuned\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.060476 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.060462    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:38.060522 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.060480    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:38.060522 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.060488    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j6hfv for pod openshift-network-diagnostics/network-check-target-92xq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:38.060593 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.060530    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv podName:6dfe72c3-55e9-4cee-916d-8e7e88301f67 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:38.560520675 +0000 UTC m=+2.007369975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j6hfv" (UniqueName: "kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv") pod "network-check-target-92xq5" (UID: "6dfe72c3-55e9-4cee-916d-8e7e88301f67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:38.069682 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.069660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwvr\" (UniqueName: \"kubernetes.io/projected/dcaae002-b584-4e0a-b423-86454a1a68cc-kube-api-access-jhwvr\") pod \"multus-l6zvc\" (UID: \"dcaae002-b584-4e0a-b423-86454a1a68cc\") " pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.070133 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.070116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxl8m\" (UniqueName: \"kubernetes.io/projected/36bf149a-2851-4d65-ac83-cd613597aae1-kube-api-access-pxl8m\") pod \"iptables-alerter-x4kps\" (UID: \"36bf149a-2851-4d65-ac83-cd613597aae1\") " pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:38.072256 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.071918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4v8q\" (UniqueName: \"kubernetes.io/projected/6921424a-8439-4602-acc4-e525c0779616-kube-api-access-r4v8q\") pod \"node-ca-wrgjj\" (UID: \"6921424a-8439-4602-acc4-e525c0779616\") " pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:38.072256 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.071950    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8vp\" (UniqueName: \"kubernetes.io/projected/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-kube-api-access-4g8vp\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:38.072256 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.072124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffdl2\" (UniqueName: \"kubernetes.io/projected/473c79ea-44eb-44fd-8f88-2ed2b11bbb15-kube-api-access-ffdl2\") pod \"tuned-298tw\" (UID: \"473c79ea-44eb-44fd-8f88-2ed2b11bbb15\") " pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.072256 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.072154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-764g5\" (UniqueName: \"kubernetes.io/projected/4ca104f8-d1bd-4ea6-b807-b05829d737fe-kube-api-access-764g5\") pod \"multus-additional-cni-plugins-sczc4\" (UID: \"4ca104f8-d1bd-4ea6-b807-b05829d737fe\") " pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.073029 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.073009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xft7k\" (UniqueName: \"kubernetes.io/projected/c4eb704f-bf54-463d-a108-fe87f61a4e44-kube-api-access-xft7k\") pod \"aws-ebs-csi-driver-node-5trm2\" (UID: \"c4eb704f-bf54-463d-a108-fe87f61a4e44\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.073243 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.073218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8q7t\" (UniqueName: \"kubernetes.io/projected/bf76e089-703b-4a88-ab06-e23f370322bf-kube-api-access-d8q7t\") pod \"ovnkube-node-xvvg4\" (UID: \"bf76e089-703b-4a88-ab06-e23f370322bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.074293 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.074276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntbp\" (UniqueName: \"kubernetes.io/projected/5edec3b9-e837-4bb1-9ec5-ffe2163f36df-kube-api-access-vntbp\") pod \"node-resolver-nxbfp\" (UID: \"5edec3b9-e837-4bb1-9ec5-ffe2163f36df\") " pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:38.087583 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.087566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:22:38.116500 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.116460    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 14:17:36 +0000 UTC" deadline="2028-05-11 08:39:43.593889692 +0000 UTC"
Jul 28 14:22:38.116500 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.116499    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15666h17m5.477393035s"
Jul 28 14:22:38.246763 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.246713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-975st"
Jul 28 14:22:38.252661 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.252642    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:38.262792 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.262774    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2"
Jul 28 14:22:38.277702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.277679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-298tw"
Jul 28 14:22:38.303282 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.303263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sczc4"
Jul 28 14:22:38.316930 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.316911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l6zvc"
Jul 28 14:22:38.332252 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.332228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nxbfp"
Jul 28 14:22:38.347831 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.347814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wrgjj"
Jul 28 14:22:38.350190 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:38.350164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195258ab55c23e56ac6696c29a5c10c0.slice/crio-166051883a4782f11682e1c2634c2ff1b0488ed44fc0b97c40f2433a4c074596 WatchSource:0}: Error finding container 166051883a4782f11682e1c2634c2ff1b0488ed44fc0b97c40f2433a4c074596: Status 404 returned error can't find the container with id 166051883a4782f11682e1c2634c2ff1b0488ed44fc0b97c40f2433a4c074596
Jul 28 14:22:38.350662 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:38.350643    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edec3b9_e837_4bb1_9ec5_ffe2163f36df.slice/crio-1c1782f3658b67c6f42f8033537bfb3b2b2d540ca584095d92429584d6a81e06 WatchSource:0}: Error finding container 1c1782f3658b67c6f42f8033537bfb3b2b2d540ca584095d92429584d6a81e06: Status 404 returned error can't find the container with id 1c1782f3658b67c6f42f8033537bfb3b2b2d540ca584095d92429584d6a81e06
Jul 28 14:22:38.355428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.355409    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:22:38.355675 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.355656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x4kps"
Jul 28 14:22:38.355873 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:38.355852    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6921424a_8439_4602_acc4_e525c0779616.slice/crio-beb950c62aee27fd5a24555f8971fc0ecbaa6c80f98a51f079c3caf5c6e80be7 WatchSource:0}: Error finding container beb950c62aee27fd5a24555f8971fc0ecbaa6c80f98a51f079c3caf5c6e80be7: Status 404 returned error can't find the container with id beb950c62aee27fd5a24555f8971fc0ecbaa6c80f98a51f079c3caf5c6e80be7
Jul 28 14:22:38.362774 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:38.362757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bf149a_2851_4d65_ac83_cd613597aae1.slice/crio-1749f2b3993935b2965177822a55b5ae7b38a3dbe400dced4ee99106f7e119c7 WatchSource:0}: Error finding container 1749f2b3993935b2965177822a55b5ae7b38a3dbe400dced4ee99106f7e119c7: Status 404 returned error can't find the container with id 1749f2b3993935b2965177822a55b5ae7b38a3dbe400dced4ee99106f7e119c7
Jul 28 14:22:38.416414 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:38.416378    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf76e089_703b_4a88_ab06_e23f370322bf.slice/crio-7ad840ee1d4060c24ff9152c0cfbc3eaa33493ae652c2bf368cd5cd3aa733fd2 WatchSource:0}: Error finding container 7ad840ee1d4060c24ff9152c0cfbc3eaa33493ae652c2bf368cd5cd3aa733fd2: Status 404 returned error can't find the container with id 7ad840ee1d4060c24ff9152c0cfbc3eaa33493ae652c2bf368cd5cd3aa733fd2
Jul 28 14:22:38.558845 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.558825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:38.558937 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.558860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:38.558984 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.558961    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:38.559023 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.559006    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs podName:ed53709b-b3b8-4ef7-a9c2-35a9789dac55 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:39.558994671 +0000 UTC m=+3.005843959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs") pod "network-metrics-daemon-h5w88" (UID: "ed53709b-b3b8-4ef7-a9c2-35a9789dac55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:38.559065 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.558961    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:38.559106 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.559087    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret podName:a00f8f97-a454-45d2-9758-e09e9f0ff6f3 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:39.559074512 +0000 UTC m=+3.005923812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret") pod "global-pull-secret-syncer-txplh" (UID: "a00f8f97-a454-45d2-9758-e09e9f0ff6f3") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:38.624086 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:38.624067    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6857bda9fd953712469ebe65794e79.slice/crio-7a35a44ea5d9427231a4110812816abd1d3a11643bde3d7a292c069e640e7df2 WatchSource:0}: Error finding container 7a35a44ea5d9427231a4110812816abd1d3a11643bde3d7a292c069e640e7df2: Status 404 returned error can't find the container with id 7a35a44ea5d9427231a4110812816abd1d3a11643bde3d7a292c069e640e7df2
Jul 28 14:22:38.646297 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:38.646279    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473c79ea_44eb_44fd_8f88_2ed2b11bbb15.slice/crio-d6ed1cb25fb6c1d6aa5cf1cb796c8937ad4334f51c244ad8357690b61d62f271 WatchSource:0}: Error finding container d6ed1cb25fb6c1d6aa5cf1cb796c8937ad4334f51c244ad8357690b61d62f271: Status 404 returned error can't find the container with id d6ed1cb25fb6c1d6aa5cf1cb796c8937ad4334f51c244ad8357690b61d62f271
Jul 28 14:22:38.659958 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.659937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:38.660062 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.660051    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:38.660103 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.660067    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:38.660103 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.660075    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j6hfv for pod openshift-network-diagnostics/network-check-target-92xq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:38.660169 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:38.660114    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv podName:6dfe72c3-55e9-4cee-916d-8e7e88301f67 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:39.660101739 +0000 UTC m=+3.106951026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6hfv" (UniqueName: "kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv") pod "network-check-target-92xq5" (UID: "6dfe72c3-55e9-4cee-916d-8e7e88301f67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:38.688902 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:22:38.688884    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca104f8_d1bd_4ea6_b807_b05829d737fe.slice/crio-c20e68de402e9ac22a748e055c8ed0d2aef5fb885fc2c66973a737ce3ebdc593 WatchSource:0}: Error finding container c20e68de402e9ac22a748e055c8ed0d2aef5fb885fc2c66973a737ce3ebdc593: Status 404 returned error can't find the container with id c20e68de402e9ac22a748e055c8ed0d2aef5fb885fc2c66973a737ce3ebdc593
Jul 28 14:22:38.903624 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:38.903542    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:39.002861 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.002801    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 14:17:37 +0000 UTC" deadline="2028-02-04 01:20:34.409263182 +0000 UTC"
Jul 28 14:22:39.002861 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.002859    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13330h57m55.406407961s"
Jul 28 14:22:39.246500 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.246426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:39.246652 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.246552    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:39.246975 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.246955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:39.247070 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.247057    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:39.247133 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.247117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:39.247195 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.247179    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:39.258501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.258455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nxbfp" event={"ID":"5edec3b9-e837-4bb1-9ec5-ffe2163f36df","Type":"ContainerStarted","Data":"1c1782f3658b67c6f42f8033537bfb3b2b2d540ca584095d92429584d6a81e06"}
Jul 28 14:22:39.265236 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.265208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal" event={"ID":"195258ab55c23e56ac6696c29a5c10c0","Type":"ContainerStarted","Data":"166051883a4782f11682e1c2634c2ff1b0488ed44fc0b97c40f2433a4c074596"}
Jul 28 14:22:39.271130 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.271105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sczc4" event={"ID":"4ca104f8-d1bd-4ea6-b807-b05829d737fe","Type":"ContainerStarted","Data":"c20e68de402e9ac22a748e055c8ed0d2aef5fb885fc2c66973a737ce3ebdc593"}
Jul 28 14:22:39.274702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.274677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-298tw" event={"ID":"473c79ea-44eb-44fd-8f88-2ed2b11bbb15","Type":"ContainerStarted","Data":"d6ed1cb25fb6c1d6aa5cf1cb796c8937ad4334f51c244ad8357690b61d62f271"}
Jul 28 14:22:39.276743 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.276694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal" event={"ID":"1a6857bda9fd953712469ebe65794e79","Type":"ContainerStarted","Data":"7a35a44ea5d9427231a4110812816abd1d3a11643bde3d7a292c069e640e7df2"}
Jul 28 14:22:39.279413 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.279378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"7ad840ee1d4060c24ff9152c0cfbc3eaa33493ae652c2bf368cd5cd3aa733fd2"}
Jul 28 14:22:39.285760 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.285727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x4kps" event={"ID":"36bf149a-2851-4d65-ac83-cd613597aae1","Type":"ContainerStarted","Data":"1749f2b3993935b2965177822a55b5ae7b38a3dbe400dced4ee99106f7e119c7"}
Jul 28 14:22:39.290202 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.290178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wrgjj" event={"ID":"6921424a-8439-4602-acc4-e525c0779616","Type":"ContainerStarted","Data":"beb950c62aee27fd5a24555f8971fc0ecbaa6c80f98a51f079c3caf5c6e80be7"}
Jul 28 14:22:39.568111 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.568079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:39.568308 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.568139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:39.568308 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.568292    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:39.568526 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.568348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret podName:a00f8f97-a454-45d2-9758-e09e9f0ff6f3 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:41.568328803 +0000 UTC m=+5.015178094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret") pod "global-pull-secret-syncer-txplh" (UID: "a00f8f97-a454-45d2-9758-e09e9f0ff6f3") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:39.568752 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.568732    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:39.568829 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.568802    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs podName:ed53709b-b3b8-4ef7-a9c2-35a9789dac55 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:41.568766065 +0000 UTC m=+5.015615357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs") pod "network-metrics-daemon-h5w88" (UID: "ed53709b-b3b8-4ef7-a9c2-35a9789dac55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:39.668527 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:39.668494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:39.668719 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.668680    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:39.668719 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.668699    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:39.668719 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.668710    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j6hfv for pod openshift-network-diagnostics/network-check-target-92xq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:39.668886 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:39.668777    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv podName:6dfe72c3-55e9-4cee-916d-8e7e88301f67 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:41.668752161 +0000 UTC m=+5.115601458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6hfv" (UniqueName: "kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv") pod "network-check-target-92xq5" (UID: "6dfe72c3-55e9-4cee-916d-8e7e88301f67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:40.003060 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:40.003020    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 14:17:37 +0000 UTC" deadline="2028-01-25 08:25:19.834064661 +0000 UTC"
Jul 28 14:22:40.003060 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:40.003058    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13098h2m39.831011361s"
Jul 28 14:22:40.032893 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:40.032863    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 14:22:40.293408 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:40.293312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-975st" event={"ID":"870f6c21-2fc5-436b-9f77-77b53329ca23","Type":"ContainerStarted","Data":"3f3744f857d68457c7411890e3b343a1a1cfb227352d0e7f2075a180567116d5"}
Jul 28 14:22:40.302043 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:40.302010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l6zvc" event={"ID":"dcaae002-b584-4e0a-b423-86454a1a68cc","Type":"ContainerStarted","Data":"55239927ecf466abb6baf3d6dbb51c84da2e1d4344c3100b1c0cd3e2bb65ff3d"}
Jul 28 14:22:40.318250 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:40.318221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2" event={"ID":"c4eb704f-bf54-463d-a108-fe87f61a4e44","Type":"ContainerStarted","Data":"6c7bc87cb3e85577b6385c3f723e600094e5aebe23f419a88c7ae26eea59fb9d"}
Jul 28 14:22:41.246693 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:41.245580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:41.246693 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.245677    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:41.246693 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:41.245941    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:41.246693 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.246004    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:41.246693 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:41.246032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:41.246693 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.246069    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:41.585875 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:41.585843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:41.586048 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:41.585923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:41.586048 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.586025    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:41.586048 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.586026    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:41.586218 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.586082    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs podName:ed53709b-b3b8-4ef7-a9c2-35a9789dac55 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:45.586063654 +0000 UTC m=+9.032912961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs") pod "network-metrics-daemon-h5w88" (UID: "ed53709b-b3b8-4ef7-a9c2-35a9789dac55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:41.586218 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.586101    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret podName:a00f8f97-a454-45d2-9758-e09e9f0ff6f3 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:45.586091373 +0000 UTC m=+9.032940664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret") pod "global-pull-secret-syncer-txplh" (UID: "a00f8f97-a454-45d2-9758-e09e9f0ff6f3") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:41.686583 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:41.686549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:41.686757 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.686700    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:41.686757 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.686719    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:41.686757 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.686731    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j6hfv for pod openshift-network-diagnostics/network-check-target-92xq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:41.686914 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:41.686788    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv podName:6dfe72c3-55e9-4cee-916d-8e7e88301f67 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:45.686767172 +0000 UTC m=+9.133616479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6hfv" (UniqueName: "kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv") pod "network-check-target-92xq5" (UID: "6dfe72c3-55e9-4cee-916d-8e7e88301f67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:43.247068 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:43.246769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:43.247068 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:43.246900    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:43.247649 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:43.247324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:43.247649 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:43.247430    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:43.247649 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:43.247574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:43.247788 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:43.247671    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:45.246423 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:45.246377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:45.246859 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:45.246383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:45.246859 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.246518    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:45.246859 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:45.246402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:45.246859 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.246605    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:45.246859 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.246753    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:45.617523 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:45.617426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:45.617523 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:45.617522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:45.617715 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.617557    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:45.617715 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.617615    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:45.617715 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.617629    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret podName:a00f8f97-a454-45d2-9758-e09e9f0ff6f3 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:53.617609425 +0000 UTC m=+17.064458724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret") pod "global-pull-secret-syncer-txplh" (UID: "a00f8f97-a454-45d2-9758-e09e9f0ff6f3") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:45.617715 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.617648    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs podName:ed53709b-b3b8-4ef7-a9c2-35a9789dac55 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:53.617637294 +0000 UTC m=+17.064486581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs") pod "network-metrics-daemon-h5w88" (UID: "ed53709b-b3b8-4ef7-a9c2-35a9789dac55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:45.717889 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:45.717846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:45.718052 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.718035    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:45.718119 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.718060    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:45.718119 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.718073    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j6hfv for pod openshift-network-diagnostics/network-check-target-92xq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:45.718229 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:45.718142    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv podName:6dfe72c3-55e9-4cee-916d-8e7e88301f67 nodeName:}" failed. No retries permitted until 2026-07-28 14:22:53.718122891 +0000 UTC m=+17.164972193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6hfv" (UniqueName: "kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv") pod "network-check-target-92xq5" (UID: "6dfe72c3-55e9-4cee-916d-8e7e88301f67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:47.246837 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:47.246565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:47.246837 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:47.246688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:47.246837 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:47.246752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:47.247873 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:47.247842    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:47.247967 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:47.247927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:47.248068 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:47.248046    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:49.245599 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:49.245557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:49.246077 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:49.245570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:49.246077 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:49.245676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:49.246077 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:49.245697    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:49.246077 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:49.245773    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:49.246077 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:49.245850    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:51.246273 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:51.246242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:51.246273 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:51.246262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:51.246703 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:51.246369    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:51.246703 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:51.246466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:51.246703 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:51.246487    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:51.246703 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:51.246553    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:53.246024 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:53.245900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:53.246024 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:53.245923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:53.246514 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.246028    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:53.246514 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.246098    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:53.246514 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:53.246119    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:53.246514 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.246217    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:53.672711 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:53.672673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:53.672873 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:53.672730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:53.672873 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.672829    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:53.672873 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.672848    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:53.673018 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.672892    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs podName:ed53709b-b3b8-4ef7-a9c2-35a9789dac55 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:09.672871703 +0000 UTC m=+33.119721010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs") pod "network-metrics-daemon-h5w88" (UID: "ed53709b-b3b8-4ef7-a9c2-35a9789dac55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:22:53.673018 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.672908    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret podName:a00f8f97-a454-45d2-9758-e09e9f0ff6f3 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:09.672901641 +0000 UTC m=+33.119750928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret") pod "global-pull-secret-syncer-txplh" (UID: "a00f8f97-a454-45d2-9758-e09e9f0ff6f3") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:22:53.773925 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:53.773899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:53.774018 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.774006    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:22:53.774061 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.774024    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:22:53.774061 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.774034    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j6hfv for pod openshift-network-diagnostics/network-check-target-92xq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:53.774128 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:53.774072    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv podName:6dfe72c3-55e9-4cee-916d-8e7e88301f67 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:09.774061782 +0000 UTC m=+33.220911069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6hfv" (UniqueName: "kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv") pod "network-check-target-92xq5" (UID: "6dfe72c3-55e9-4cee-916d-8e7e88301f67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:22:55.245894 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:55.245854    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:55.246475 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:55.245898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:55.246475 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:55.245870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:55.246475 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:55.245998    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:55.246475 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:55.246109    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:55.246475 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:55.246203    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:57.246942 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:57.246909    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:57.247433 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:57.247013    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:57.247433 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:57.247028    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:57.247433 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:57.247107    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:57.247433 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:57.247130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:57.247433 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:57.247197    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:58.358811 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:58.358497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal" event={"ID":"1a6857bda9fd953712469ebe65794e79","Type":"ContainerStarted","Data":"c4581beb9d374006643da4d3f9496f833f17dd15e2480e4ebacf4daac0e5b22b"}
Jul 28 14:22:58.374076 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:58.373832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l6zvc" event={"ID":"dcaae002-b584-4e0a-b423-86454a1a68cc","Type":"ContainerStarted","Data":"e8bdeb4a7af370208b8ed94952a1dbfc1a7686017bff28a5b4d88ceaf23ca959"}
Jul 28 14:22:58.374854 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:58.374812    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-160.ec2.internal" podStartSLOduration=21.374794173 podStartE2EDuration="21.374794173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:58.373849734 +0000 UTC m=+21.820699043" watchObservedRunningTime="2026-07-28 14:22:58.374794173 +0000 UTC m=+21.821643483"
Jul 28 14:22:58.376349 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:58.376295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-298tw" event={"ID":"473c79ea-44eb-44fd-8f88-2ed2b11bbb15","Type":"ContainerStarted","Data":"9d7f78c50bc88f38d020bf35eff648398e5b68092b5396a404a33b882f571fdc"}
Jul 28 14:22:58.378868 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:58.378838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"ef759b4ed14ffc1a68b53bc5e464803f4fda43e90f0811c2d90c82a6026fc387"}
Jul 28 14:22:58.413282 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:58.413232    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l6zvc" podStartSLOduration=2.8030522060000003 podStartE2EDuration="21.413206757s" totalImagesPullingTime="18.610154551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:58.392978779 +0000 UTC m=+21.839828089" watchObservedRunningTime="2026-07-28 14:22:58.413206757 +0000 UTC m=+21.860056066"
Jul 28 14:22:58.414085 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:58.414045    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-298tw" podStartSLOduration=2.183681906 podStartE2EDuration="21.414033285s" totalImagesPullingTime="19.230351379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:58.413208355 +0000 UTC m=+21.860057667" watchObservedRunningTime="2026-07-28 14:22:58.414033285 +0000 UTC m=+21.860882595"
Jul 28 14:22:59.246017 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.245929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:22:59.246017 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.245929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:22:59.246192 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:59.246035    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:22:59.246192 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.246065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:22:59.246192 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:59.246159    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:22:59.246348 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:22:59.246228    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:22:59.382183 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.382152    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ca104f8-d1bd-4ea6-b807-b05829d737fe" containerID="fe2667577d9495d40684d3c869fcd8e4764a01e3dd03395219972629c0815a9f" exitCode=0
Jul 28 14:22:59.382807 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.382250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sczc4" event={"ID":"4ca104f8-d1bd-4ea6-b807-b05829d737fe","Type":"ContainerDied","Data":"fe2667577d9495d40684d3c869fcd8e4764a01e3dd03395219972629c0815a9f"}
Jul 28 14:22:59.384300 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.383908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x4kps" event={"ID":"36bf149a-2851-4d65-ac83-cd613597aae1","Type":"ContainerStarted","Data":"590fb058edc5b3711d451a8e9962b48ffb8544d5ecd1382a1d8e3fa4225f5a6b"}
Jul 28 14:22:59.385903 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.385871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-975st" event={"ID":"870f6c21-2fc5-436b-9f77-77b53329ca23","Type":"ContainerStarted","Data":"f881aac5635bf7bf84f585bba524954ad718cb67ef2df89cf94724a59805351b"}
Jul 28 14:22:59.389145 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.389126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"a33591561d7e62fc817ab8fa787bb8d92ff125e2f5d5db33f08a9faebd51ebfb"}
Jul 28 14:22:59.389242 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.389186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"a08d264228bce0a86809de4bbe8d936d8bb76624bb33aaef39a9ce6ea2c2b71d"}
Jul 28 14:22:59.389242 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.389219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"c76b9bb46f09ff43d2490796765f8672cfac18f568d5442bb0910c0046e34682"}
Jul 28 14:22:59.389350 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.389260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"adf547571da4038879d00f667cc28ae5eda8b70ccd807013b7f4897319504c0e"}
Jul 28 14:22:59.389350 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.389294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"8cab24474c9d76a2f6701a6af903b2823c6a648962bcd2d5491de043e33c2696"}
Jul 28 14:22:59.390671 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.390651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wrgjj" event={"ID":"6921424a-8439-4602-acc4-e525c0779616","Type":"ContainerStarted","Data":"9efc647cb87dbd4ed774f5c27db1436d01c95cc58ebaf53a502da22259a20dfd"}
Jul 28 14:22:59.392165 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.392144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nxbfp" event={"ID":"5edec3b9-e837-4bb1-9ec5-ffe2163f36df","Type":"ContainerStarted","Data":"31287d2743fd0f58ceaa5358308702076e9890004f5216ed8b68e4b8c3d31260"}
Jul 28 14:22:59.395933 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.395913    2579 generic.go:356] "Generic (PLEG): container finished" podID="195258ab55c23e56ac6696c29a5c10c0" containerID="8c2caa2ffbd6f48dacc6adbeca3d943f62ef7bae7043766c9c50266752af80fa" exitCode=0
Jul 28 14:22:59.396023 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.395975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal" event={"ID":"195258ab55c23e56ac6696c29a5c10c0","Type":"ContainerDied","Data":"8c2caa2ffbd6f48dacc6adbeca3d943f62ef7bae7043766c9c50266752af80fa"}
Jul 28 14:22:59.397492 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.397463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2" event={"ID":"c4eb704f-bf54-463d-a108-fe87f61a4e44","Type":"ContainerStarted","Data":"da5aac2c6dcabc4f4417353282d8e0c03f43dfdd79133fcb89afad0d2fdd7d2c"}
Jul 28 14:22:59.436587 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.436553    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nxbfp" podStartSLOduration=2.954053961 podStartE2EDuration="22.436539326s" totalImagesPullingTime="19.482485365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:59.436119614 +0000 UTC m=+22.882968923" watchObservedRunningTime="2026-07-28 14:22:59.436539326 +0000 UTC m=+22.883388635"
Jul 28 14:22:59.467988 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.467948    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wrgjj" podStartSLOduration=3.001331731 podStartE2EDuration="22.467936428s" totalImagesPullingTime="19.466604697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:59.452211759 +0000 UTC m=+22.899061067" watchObservedRunningTime="2026-07-28 14:22:59.467936428 +0000 UTC m=+22.914785736"
Jul 28 14:22:59.484893 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.484855    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-975st" podStartSLOduration=4.665481382 podStartE2EDuration="22.484841762s" totalImagesPullingTime="17.81936038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:59.467626295 +0000 UTC m=+22.914475601" watchObservedRunningTime="2026-07-28 14:22:59.484841762 +0000 UTC m=+22.931691096"
Jul 28 14:22:59.485382 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.485355    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x4kps" podStartSLOduration=2.972812521 podStartE2EDuration="22.485342921s" totalImagesPullingTime="19.5125304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:59.484963621 +0000 UTC m=+22.931812908" watchObservedRunningTime="2026-07-28 14:22:59.485342921 +0000 UTC m=+22.932192229"
Jul 28 14:22:59.906893 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:22:59.906855    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 14:23:00.153006 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:00.152872    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T14:22:59.90687586Z","UUID":"170c32a1-e406-4e66-aab2-7e44e21bdcb9","Handler":null,"Name":"","Endpoint":""}
Jul 28 14:23:00.154474 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:00.154453    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 14:23:00.154581 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:00.154490    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 14:23:00.402384 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:00.402123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal" event={"ID":"195258ab55c23e56ac6696c29a5c10c0","Type":"ContainerStarted","Data":"5feda543a4bbaceb8cd2293b2ecb3a949984c28d35f412e8c37518e29a0a868b"}
Jul 28 14:23:00.404828 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:00.404764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2" event={"ID":"c4eb704f-bf54-463d-a108-fe87f61a4e44","Type":"ContainerStarted","Data":"309e8d8bc85b484fcb18b8711837ccb53b4190318ba4b391cf585e0559810136"}
Jul 28 14:23:00.428272 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:00.428217    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-160.ec2.internal" podStartSLOduration=23.428202068 podStartE2EDuration="23.428202068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:00.427828989 +0000 UTC m=+23.874678302" watchObservedRunningTime="2026-07-28 14:23:00.428202068 +0000 UTC m=+23.875051399"
Jul 28 14:23:01.246355 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:01.246199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:01.246355 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:01.246216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:01.246355 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:01.246323    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:23:01.246593 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:01.246423    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:23:01.246593 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:01.246466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:01.246593 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:01.246577    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:23:01.411867 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:01.411784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"1a988554dc89f7fbdefa7ccaa3d54f939b0fd181eb0d98a70bd2915836102eea"}
Jul 28 14:23:01.414284 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:01.414236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2" event={"ID":"c4eb704f-bf54-463d-a108-fe87f61a4e44","Type":"ContainerStarted","Data":"62fb26ea0a9af4718289cb8c6af61c62731000967114d79eb5760664c36102cf"}
Jul 28 14:23:01.436639 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:01.436588    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5trm2" podStartSLOduration=8.159649354 podStartE2EDuration="24.436568644s" totalImagesPullingTime="16.27691929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:01.43647843 +0000 UTC m=+24.883327740" watchObservedRunningTime="2026-07-28 14:23:01.436568644 +0000 UTC m=+24.883417955"
Jul 28 14:23:02.483027 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:02.482996    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-975st"
Jul 28 14:23:02.483691 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:02.483666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-975st"
Jul 28 14:23:03.246277 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:03.246241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:03.246473 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:03.246293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:03.246473 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:03.246247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:03.246473 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:03.246379    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:23:03.246473 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:03.246460    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:23:03.246649 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:03.246545    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:23:03.418300 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:03.418268    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-975st"
Jul 28 14:23:03.418787 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:03.418763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-975st"
Jul 28 14:23:04.422832 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.422651    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ca104f8-d1bd-4ea6-b807-b05829d737fe" containerID="54e9f5d3f08f604722d003f6537579ea79af5135fa4057e326e97f15d89ebf74" exitCode=0
Jul 28 14:23:04.423510 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.422738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sczc4" event={"ID":"4ca104f8-d1bd-4ea6-b807-b05829d737fe","Type":"ContainerDied","Data":"54e9f5d3f08f604722d003f6537579ea79af5135fa4057e326e97f15d89ebf74"}
Jul 28 14:23:04.426896 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.426874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" event={"ID":"bf76e089-703b-4a88-ab06-e23f370322bf","Type":"ContainerStarted","Data":"88901c20225c796473e93806f69097908228bd82d30fe33e7a4dd6a9567b765c"}
Jul 28 14:23:04.427140 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.427118    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:23:04.427240 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.427196    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:23:04.442851 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.442822    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:23:04.442999 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.442925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:23:04.480632 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.480577    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4" podStartSLOduration=7.687431753 podStartE2EDuration="27.480561515s" totalImagesPullingTime="19.793129762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:04.480296303 +0000 UTC m=+27.927145611" watchObservedRunningTime="2026-07-28 14:23:04.480561515 +0000 UTC m=+27.927410824"
Jul 28 14:23:04.797267 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:04.797153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:23:05.250657 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.250629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:05.250782 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.250662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:05.250782 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.250662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:05.250782 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:05.250760    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:23:05.250970 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:05.250853    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:23:05.250970 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:05.250952    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:23:05.430388 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.430311    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ca104f8-d1bd-4ea6-b807-b05829d737fe" containerID="96c43e1150f087f6446f19fad9ef35aa47a8017c5340b12b5b43bea20963de7a" exitCode=0
Jul 28 14:23:05.430859 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.430416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sczc4" event={"ID":"4ca104f8-d1bd-4ea6-b807-b05829d737fe","Type":"ContainerDied","Data":"96c43e1150f087f6446f19fad9ef35aa47a8017c5340b12b5b43bea20963de7a"}
Jul 28 14:23:05.517193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.517169    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txplh"]
Jul 28 14:23:05.517327 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.517246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:05.517412 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:05.517323    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:23:05.520437 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.520414    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-92xq5"]
Jul 28 14:23:05.520557 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.520488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:05.520624 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:05.520602    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:23:05.520998 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.520976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h5w88"]
Jul 28 14:23:05.521085 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:05.521071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:05.521165 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:05.521150    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:23:06.434956 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:06.434774    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ca104f8-d1bd-4ea6-b807-b05829d737fe" containerID="9c2674420cd79fd3b819b2b405a6cdf2531cb1238f89b0de91a5d1722ca5b04f" exitCode=0
Jul 28 14:23:06.435599 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:06.434858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sczc4" event={"ID":"4ca104f8-d1bd-4ea6-b807-b05829d737fe","Type":"ContainerDied","Data":"9c2674420cd79fd3b819b2b405a6cdf2531cb1238f89b0de91a5d1722ca5b04f"}
Jul 28 14:23:07.248424 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:07.248283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:07.249693 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:07.248621    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:23:07.249693 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:07.249068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:07.249693 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:07.249188    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:23:07.249693 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:07.249213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:07.249693 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:07.249340    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:23:09.246068 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:09.245992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:09.246068 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:09.246015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:09.246068 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:09.246027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:09.246685 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.246121    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-92xq5" podUID="6dfe72c3-55e9-4cee-916d-8e7e88301f67"
Jul 28 14:23:09.246685 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.246240    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-txplh" podUID="a00f8f97-a454-45d2-9758-e09e9f0ff6f3"
Jul 28 14:23:09.246685 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.246333    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5w88" podUID="ed53709b-b3b8-4ef7-a9c2-35a9789dac55"
Jul 28 14:23:09.697236 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:09.697206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:09.697452 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:09.697291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:09.697452 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.697362    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 14:23:09.697452 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.697373    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:23:09.697452 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.697443    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs podName:ed53709b-b3b8-4ef7-a9c2-35a9789dac55 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:41.697426925 +0000 UTC m=+65.144276212 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs") pod "network-metrics-daemon-h5w88" (UID: "ed53709b-b3b8-4ef7-a9c2-35a9789dac55") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 14:23:09.697452 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.697458    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret podName:a00f8f97-a454-45d2-9758-e09e9f0ff6f3 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:41.697452235 +0000 UTC m=+65.144301522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret") pod "global-pull-secret-syncer-txplh" (UID: "a00f8f97-a454-45d2-9758-e09e9f0ff6f3") : object "kube-system"/"original-pull-secret" not registered
Jul 28 14:23:09.797685 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:09.797650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:09.797859 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.797846    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 14:23:09.797927 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.797870    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 14:23:09.797927 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.797883    2579 projected.go:196] Error preparing data for projected volume kube-api-access-j6hfv for pod openshift-network-diagnostics/network-check-target-92xq5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:23:09.798021 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:09.797948    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv podName:6dfe72c3-55e9-4cee-916d-8e7e88301f67 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:41.7979286 +0000 UTC m=+65.244777888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6hfv" (UniqueName: "kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv") pod "network-check-target-92xq5" (UID: "6dfe72c3-55e9-4cee-916d-8e7e88301f67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 14:23:10.904356 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.904286    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-160.ec2.internal" event="NodeReady"
Jul 28 14:23:10.904830 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.904429    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 14:23:10.942539 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.942514    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"]
Jul 28 14:23:10.960223 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.960200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"]
Jul 28 14:23:10.960340 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.960319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"
Jul 28 14:23:10.965190 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.965136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 14:23:10.965738 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.965716    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 14:23:10.965851 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.965754    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 14:23:10.965928 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.965904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-44dpc\""
Jul 28 14:23:10.966087 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.966069    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 14:23:10.982334 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.982313    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dc8c84dc8-6lv4c"]
Jul 28 14:23:10.982450 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.982435    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:10.984947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.984928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 14:23:10.985046 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.985001    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 14:23:10.985289 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.985273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 14:23:10.985509 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.985494    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 14:23:10.996444 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.996426    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"]
Jul 28 14:23:10.996537 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.996524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:10.999091 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.999074    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 14:23:10.999947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.999934    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 14:23:10.999947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:10.999942    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xg5kp\""
Jul 28 14:23:11.000186 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.000172    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 14:23:11.004596 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.004577    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 14:23:11.010488 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.010473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"]
Jul 28 14:23:11.010547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.010494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"]
Jul 28 14:23:11.010547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.010507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dc8c84dc8-6lv4c"]
Jul 28 14:23:11.010547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.010519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"]
Jul 28 14:23:11.010637 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.010567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.010637 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.010601    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-px49m"]
Jul 28 14:23:11.014195 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.014178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 14:23:11.024618 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.024603    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kjgps"]
Jul 28 14:23:11.024720 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.024706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.027349 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.027334    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pl2dt\""
Jul 28 14:23:11.027454 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.027354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 14:23:11.027454 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.027363    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 14:23:11.038629 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.038610    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kjgps"]
Jul 28 14:23:11.038707 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.038638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-px49m"]
Jul 28 14:23:11.038745 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.038711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kjgps"
Jul 28 14:23:11.041053 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.041036    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 14:23:11.041114 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.041075    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 14:23:11.041336 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.041323    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-htnvc\""
Jul 28 14:23:11.041627 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.041610    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 14:23:11.107241 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22fw\" (UniqueName: \"kubernetes.io/projected/de66e80c-b431-485e-bc0b-ea3eede95805-kube-api-access-m22fw\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.107320 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-image-registry-private-configuration\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.107320 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-klusterlet-config\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.107320 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/de66e80c-b431-485e-bc0b-ea3eede95805-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.107320 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/233e6a1c-28cf-4e70-9714-efbadb1331d9-ca-trust-extracted\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.107460 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mv8m\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-kube-api-access-8mv8m\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.107460 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-hub\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.107460 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b9778bc74-7cz8t\" (UID: \"f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"
Jul 28 14:23:11.107460 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-bound-sa-token\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.107460 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.107607 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczrm\" (UniqueName: \"kubernetes.io/projected/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-kube-api-access-mczrm\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.107607 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-trusted-ca\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.107607 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-certificates\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.107607 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5l72\" (UniqueName: \"kubernetes.io/projected/904fca85-96bf-4856-b032-c57d0ab7cfbc-kube-api-access-m5l72\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.107607 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/904fca85-96bf-4856-b032-c57d0ab7cfbc-config-volume\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.107607 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.107767 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-ca\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.107767 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmdwq\" (UniqueName: \"kubernetes.io/projected/f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3-kube-api-access-rmdwq\") pod \"managed-serviceaccount-addon-agent-b9778bc74-7cz8t\" (UID: \"f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"
Jul 28 14:23:11.107767 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.107767 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.107767 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-installation-pull-secrets\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.107767 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-tmp\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.107939 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.107778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904fca85-96bf-4856-b032-c57d0ab7cfbc-tmp-dir\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.159525 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.159471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:11.208859 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.208838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-trusted-ca\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.208929 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.208866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5979f987-8fc4-42fc-9729-6db64e0c42bd-cert\") pod \"ingress-canary-kjgps\" (UID: \"5979f987-8fc4-42fc-9729-6db64e0c42bd\") " pod="openshift-ingress-canary/ingress-canary-kjgps"
Jul 28 14:23:11.208929 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.208883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-certificates\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.208929 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.208907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5l72\" (UniqueName: \"kubernetes.io/projected/904fca85-96bf-4856-b032-c57d0ab7cfbc-kube-api-access-m5l72\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.209058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.208933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/904fca85-96bf-4856-b032-c57d0ab7cfbc-config-volume\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.209058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.208956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.209058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.208988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx2rp\" (UniqueName: \"kubernetes.io/projected/5979f987-8fc4-42fc-9729-6db64e0c42bd-kube-api-access-bx2rp\") pod \"ingress-canary-kjgps\" (UID: \"5979f987-8fc4-42fc-9729-6db64e0c42bd\") " pod="openshift-ingress-canary/ingress-canary-kjgps"
Jul 28 14:23:11.209058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-ca\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.209058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmdwq\" (UniqueName: \"kubernetes.io/projected/f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3-kube-api-access-rmdwq\") pod \"managed-serviceaccount-addon-agent-b9778bc74-7cz8t\" (UID: \"f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"
Jul 28 14:23:11.209288 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.209117    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:11.209288 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.209182    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls podName:904fca85-96bf-4856-b032-c57d0ab7cfbc nodeName:}" failed. No retries permitted until 2026-07-28 14:23:11.709163881 +0000 UTC m=+35.156013187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls") pod "dns-default-px49m" (UID: "904fca85-96bf-4856-b032-c57d0ab7cfbc") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:11.209288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.209288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.209288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-installation-pull-secrets\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-tmp\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904fca85-96bf-4856-b032-c57d0ab7cfbc-tmp-dir\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22fw\" (UniqueName: \"kubernetes.io/projected/de66e80c-b431-485e-bc0b-ea3eede95805-kube-api-access-m22fw\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-image-registry-private-configuration\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-klusterlet-config\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/de66e80c-b431-485e-bc0b-ea3eede95805-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/233e6a1c-28cf-4e70-9714-efbadb1331d9-ca-trust-extracted\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mv8m\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-kube-api-access-8mv8m\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-hub\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b9778bc74-7cz8t\" (UID: \"f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"
Jul 28 14:23:11.209559 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-bound-sa-token\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-certificates\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczrm\" (UniqueName: \"kubernetes.io/projected/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-kube-api-access-mczrm\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/904fca85-96bf-4856-b032-c57d0ab7cfbc-config-volume\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904fca85-96bf-4856-b032-c57d0ab7cfbc-tmp-dir\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.209786    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.209799    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc8c84dc8-6lv4c: secret "image-registry-tls" not found
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.209851    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls podName:233e6a1c-28cf-4e70-9714-efbadb1331d9 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:11.709835142 +0000 UTC m=+35.156684432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls") pod "image-registry-dc8c84dc8-6lv4c" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9") : secret "image-registry-tls" not found
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-trusted-ca\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.210078 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.209947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-tmp\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.210848 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.210678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/de66e80c-b431-485e-bc0b-ea3eede95805-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.210963 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.210943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/233e6a1c-28cf-4e70-9714-efbadb1331d9-ca-trust-extracted\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.214739 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.214618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-installation-pull-secrets\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.214739 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.214625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.214862 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.214741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-klusterlet-config\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.214862 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.214746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-b9778bc74-7cz8t\" (UID: \"f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"
Jul 28 14:23:11.214862 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.214701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-hub\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.214862 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.214637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-image-registry-private-configuration\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.214992 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.214927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-ca\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.215161 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.215142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de66e80c-b431-485e-bc0b-ea3eede95805-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.219975 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.219946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5l72\" (UniqueName: \"kubernetes.io/projected/904fca85-96bf-4856-b032-c57d0ab7cfbc-kube-api-access-m5l72\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.221619 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.221589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmdwq\" (UniqueName: \"kubernetes.io/projected/f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3-kube-api-access-rmdwq\") pod \"managed-serviceaccount-addon-agent-b9778bc74-7cz8t\" (UID: \"f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"
Jul 28 14:23:11.221723 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.221645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mv8m\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-kube-api-access-8mv8m\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.221723 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.221701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22fw\" (UniqueName: \"kubernetes.io/projected/de66e80c-b431-485e-bc0b-ea3eede95805-kube-api-access-m22fw\") pod \"cluster-proxy-proxy-agent-b9457d8b5-w6tsr\" (UID: \"de66e80c-b431-485e-bc0b-ea3eede95805\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.221812 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.221744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-bound-sa-token\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.222137 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.222117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczrm\" (UniqueName: \"kubernetes.io/projected/20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97-kube-api-access-mczrm\") pod \"klusterlet-addon-workmgr-9c6b8bd56-gq7n9\" (UID: \"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.246079 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.246056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:11.246172 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.246060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:11.246314 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.246295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:11.247622 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.247598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:11.248959 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.248834    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kq7zv\""
Jul 28 14:23:11.248959 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.248865    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 14:23:11.248959 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.248925    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 14:23:11.249163 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.249139    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 14:23:11.249403 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.249372    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 14:23:11.251070 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.249589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z6ktw\""
Jul 28 14:23:11.310072 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.310055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5979f987-8fc4-42fc-9729-6db64e0c42bd-cert\") pod \"ingress-canary-kjgps\" (UID: \"5979f987-8fc4-42fc-9729-6db64e0c42bd\") " pod="openshift-ingress-canary/ingress-canary-kjgps"
Jul 28 14:23:11.310160 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.310090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx2rp\" (UniqueName: \"kubernetes.io/projected/5979f987-8fc4-42fc-9729-6db64e0c42bd-kube-api-access-bx2rp\") pod \"ingress-canary-kjgps\" (UID: \"5979f987-8fc4-42fc-9729-6db64e0c42bd\") " pod="openshift-ingress-canary/ingress-canary-kjgps"
Jul 28 14:23:11.312197 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.312180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5979f987-8fc4-42fc-9729-6db64e0c42bd-cert\") pod \"ingress-canary-kjgps\" (UID: \"5979f987-8fc4-42fc-9729-6db64e0c42bd\") " pod="openshift-ingress-canary/ingress-canary-kjgps"
Jul 28 14:23:11.318817 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.318799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx2rp\" (UniqueName: \"kubernetes.io/projected/5979f987-8fc4-42fc-9729-6db64e0c42bd-kube-api-access-bx2rp\") pod \"ingress-canary-kjgps\" (UID: \"5979f987-8fc4-42fc-9729-6db64e0c42bd\") " pod="openshift-ingress-canary/ingress-canary-kjgps"
Jul 28 14:23:11.333697 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.333681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"
Jul 28 14:23:11.339849 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.339823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:23:11.363110 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.363092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:11.373878 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.373860    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kjgps"
Jul 28 14:23:11.454461 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.454279    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ca104f8-d1bd-4ea6-b807-b05829d737fe" containerID="379759ce73354e246b0e361f10bdeb47b1ec0132ec81d20dd89da950744d4884" exitCode=0
Jul 28 14:23:11.454461 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.454335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sczc4" event={"ID":"4ca104f8-d1bd-4ea6-b807-b05829d737fe","Type":"ContainerDied","Data":"379759ce73354e246b0e361f10bdeb47b1ec0132ec81d20dd89da950744d4884"}
Jul 28 14:23:11.570749 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.570031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"]
Jul 28 14:23:11.570749 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.570088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t"]
Jul 28 14:23:11.572413 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.572287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"]
Jul 28 14:23:11.604050 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:23:11.604014    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ad4c4c_b4ac_4c57_b017_b6d2e4d7cc97.slice/crio-923574246c7ef9c56cc319f604fb1f9a49b246422bbfa4e1704aaca10c264151 WatchSource:0}: Error finding container 923574246c7ef9c56cc319f604fb1f9a49b246422bbfa4e1704aaca10c264151: Status 404 returned error can't find the container with id 923574246c7ef9c56cc319f604fb1f9a49b246422bbfa4e1704aaca10c264151
Jul 28 14:23:11.629534 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.629506    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kjgps"]
Jul 28 14:23:11.633989 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:23:11.633960    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5979f987_8fc4_42fc_9729_6db64e0c42bd.slice/crio-91a2094bb39f54e3c076c216d34021565cf9303b7e1621f1de54692bdf8f51db WatchSource:0}: Error finding container 91a2094bb39f54e3c076c216d34021565cf9303b7e1621f1de54692bdf8f51db: Status 404 returned error can't find the container with id 91a2094bb39f54e3c076c216d34021565cf9303b7e1621f1de54692bdf8f51db
Jul 28 14:23:11.713873 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.713716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:11.713983 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.713853    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 14:23:11.713983 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.713947    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc8c84dc8-6lv4c: secret "image-registry-tls" not found
Jul 28 14:23:11.714092 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:11.713978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:11.714092 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.713991    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls podName:233e6a1c-28cf-4e70-9714-efbadb1331d9 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:12.713976221 +0000 UTC m=+36.160825508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls") pod "image-registry-dc8c84dc8-6lv4c" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9") : secret "image-registry-tls" not found
Jul 28 14:23:11.714092 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.714073    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:11.714217 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:11.714131    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls podName:904fca85-96bf-4856-b032-c57d0ab7cfbc nodeName:}" failed. No retries permitted until 2026-07-28 14:23:12.71411497 +0000 UTC m=+36.160964265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls") pod "dns-default-px49m" (UID: "904fca85-96bf-4856-b032-c57d0ab7cfbc") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:12.463713 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.463412    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ca104f8-d1bd-4ea6-b807-b05829d737fe" containerID="974d358dcc7eabf782d6c8e7875ac81dffd2ea1bacf65db1623c7f40c2c64da3" exitCode=0
Jul 28 14:23:12.463713 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.463526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sczc4" event={"ID":"4ca104f8-d1bd-4ea6-b807-b05829d737fe","Type":"ContainerDied","Data":"974d358dcc7eabf782d6c8e7875ac81dffd2ea1bacf65db1623c7f40c2c64da3"}
Jul 28 14:23:12.466221 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.466193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kjgps" event={"ID":"5979f987-8fc4-42fc-9729-6db64e0c42bd","Type":"ContainerStarted","Data":"91a2094bb39f54e3c076c216d34021565cf9303b7e1621f1de54692bdf8f51db"}
Jul 28 14:23:12.467722 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.467674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9" event={"ID":"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97","Type":"ContainerStarted","Data":"923574246c7ef9c56cc319f604fb1f9a49b246422bbfa4e1704aaca10c264151"}
Jul 28 14:23:12.469747 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.469709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t" event={"ID":"f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3","Type":"ContainerStarted","Data":"e1260b998a8bb4422a7398038b5a61ff55feb5a3044a48b2937a3dc4694aba98"}
Jul 28 14:23:12.471142 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.471113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" event={"ID":"de66e80c-b431-485e-bc0b-ea3eede95805","Type":"ContainerStarted","Data":"25504c2cf7cdb125aa304b95635635d1c83ea2a1cd2841b8d00e4de7419770f1"}
Jul 28 14:23:12.721972 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.721899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:12.721972 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.721964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:12.722159 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:12.722117    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 14:23:12.722159 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:12.722133    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc8c84dc8-6lv4c: secret "image-registry-tls" not found
Jul 28 14:23:12.722265 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:12.722195    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls podName:233e6a1c-28cf-4e70-9714-efbadb1331d9 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:14.722174785 +0000 UTC m=+38.169024094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls") pod "image-registry-dc8c84dc8-6lv4c" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9") : secret "image-registry-tls" not found
Jul 28 14:23:12.722328 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:12.722268    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:12.722328 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:12.722305    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls podName:904fca85-96bf-4856-b032-c57d0ab7cfbc nodeName:}" failed. No retries permitted until 2026-07-28 14:23:14.722293466 +0000 UTC m=+38.169142760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls") pod "dns-default-px49m" (UID: "904fca85-96bf-4856-b032-c57d0ab7cfbc") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:12.929292 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:12.929209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:13.025590 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.025362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:13.098838 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.098788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:13.178133 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.178076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:13.277726 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.277515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:13.491329 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.490540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sczc4" event={"ID":"4ca104f8-d1bd-4ea6-b807-b05829d737fe","Type":"ContainerStarted","Data":"b256af9900f1add5af184a3f12160f70d9f76ebb317f8fb2ce0758918660292a"}
Jul 28 14:23:13.517919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.517423    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sczc4" podStartSLOduration=11.534714158 podStartE2EDuration="36.517405756s" totalImagesPullingTime="24.982691598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:13.515892402 +0000 UTC m=+36.962741723" watchObservedRunningTime="2026-07-28 14:23:13.517405756 +0000 UTC m=+36.964255063"
Jul 28 14:23:13.664383 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.664323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:13.765969 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.765880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:13.865745 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.865660    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:13.966293 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:13.966166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:14.077175 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:14.077143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:14.439718 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:14.439618    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:14.517697 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:14.517611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:14.741276 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:14.740459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:14.741276 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:14.740529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:14.741276 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:14.740680    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 14:23:14.741276 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:14.740696    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc8c84dc8-6lv4c: secret "image-registry-tls" not found
Jul 28 14:23:14.741276 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:14.740755    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls podName:233e6a1c-28cf-4e70-9714-efbadb1331d9 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:18.740735898 +0000 UTC m=+42.187585191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls") pod "image-registry-dc8c84dc8-6lv4c" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9") : secret "image-registry-tls" not found
Jul 28 14:23:14.741276 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:14.741161    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:14.741276 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:14.741211    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls podName:904fca85-96bf-4856-b032-c57d0ab7cfbc nodeName:}" failed. No retries permitted until 2026-07-28 14:23:18.741197276 +0000 UTC m=+42.188046564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls") pod "dns-default-px49m" (UID: "904fca85-96bf-4856-b032-c57d0ab7cfbc") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:16.741861 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:16.741826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:16.836519 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:16.836486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:18.244521 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:18.244482    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:18.774319 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:18.774288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:18.774523 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:18.774340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:18.774523 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:18.774481    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:18.774523 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:18.774495    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 14:23:18.774523 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:18.774509    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc8c84dc8-6lv4c: secret "image-registry-tls" not found
Jul 28 14:23:18.774684 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:18.774560    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls podName:904fca85-96bf-4856-b032-c57d0ab7cfbc nodeName:}" failed. No retries permitted until 2026-07-28 14:23:26.774538259 +0000 UTC m=+50.221387562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls") pod "dns-default-px49m" (UID: "904fca85-96bf-4856-b032-c57d0ab7cfbc") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:18.774684 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:18.774584    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls podName:233e6a1c-28cf-4e70-9714-efbadb1331d9 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:26.774572881 +0000 UTC m=+50.221422189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls") pod "image-registry-dc8c84dc8-6lv4c" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9") : secret "image-registry-tls" not found
Jul 28 14:23:20.512105 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.512071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kjgps" event={"ID":"5979f987-8fc4-42fc-9729-6db64e0c42bd","Type":"ContainerStarted","Data":"d97a400809d71ee2a45db4457df65857d4d900de87c429786edf5b0a46899e4a"}
Jul 28 14:23:20.513586 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.513562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9" event={"ID":"20ad4c4c-b4ac-4c57-b017-b6d2e4d7cc97","Type":"ContainerStarted","Data":"b84a6d5470d645ea7dfe746a9e817fc6d00ff017b661a6555eaf5c6a14ceeca3"}
Jul 28 14:23:20.513718 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.513697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:20.514998 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.514973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t" event={"ID":"f1370f0c-7667-4d48-bf38-f1bd7f5fb0d3","Type":"ContainerStarted","Data":"757c631cd74fcfdf9060e11855cc6bc143bbae1e33bef291ff2adecc362b81d4"}
Jul 28 14:23:20.515914 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.515894    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9"
Jul 28 14:23:20.517335 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.517313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" event={"ID":"de66e80c-b431-485e-bc0b-ea3eede95805","Type":"ContainerStarted","Data":"d61316212debd0375df13d5dd170ef1fd57da6bc788ffbc70853ca83829d7d2e"}
Jul 28 14:23:20.517427 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.517344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" event={"ID":"de66e80c-b431-485e-bc0b-ea3eede95805","Type":"ContainerStarted","Data":"638edcc21c1e71cb20f3b8ba213bd89489fc15dea0f7cf8e2db551d7e7eb858d"}
Jul 28 14:23:20.517427 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.517355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" event={"ID":"de66e80c-b431-485e-bc0b-ea3eede95805","Type":"ContainerStarted","Data":"e956c40eef513a756c759461bae72226fe9629bb504e3be08ea95aedfaddf344"}
Jul 28 14:23:20.528959 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.528921    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kjgps" podStartSLOduration=2.507385856 podStartE2EDuration="10.52890809s" totalImagesPullingTime="8.021522234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:20.527801462 +0000 UTC m=+43.974650771" watchObservedRunningTime="2026-07-28 14:23:20.52890809 +0000 UTC m=+43.975757399"
Jul 28 14:23:20.544012 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.543963    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9c6b8bd56-gq7n9" podStartSLOduration=33.4927008 podStartE2EDuration="41.543946357s" totalImagesPullingTime="8.051245557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:20.542989913 +0000 UTC m=+43.989839247" watchObservedRunningTime="2026-07-28 14:23:20.543946357 +0000 UTC m=+43.990795667"
Jul 28 14:23:20.561946 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.561910    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-b9778bc74-7cz8t" podStartSLOduration=33.496123553 podStartE2EDuration="41.56189866s" totalImagesPullingTime="8.065775107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:20.560948697 +0000 UTC m=+44.007798006" watchObservedRunningTime="2026-07-28 14:23:20.56189866 +0000 UTC m=+44.008747968"
Jul 28 14:23:20.579859 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:20.579824    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" podStartSLOduration=33.49368738 podStartE2EDuration="41.579812133s" totalImagesPullingTime="8.086124753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:20.579562601 +0000 UTC m=+44.026411910" watchObservedRunningTime="2026-07-28 14:23:20.579812133 +0000 UTC m=+44.026661442"
Jul 28 14:23:21.218612 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:21.218579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:21.319110 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:21.319078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:22.019519 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:22.019480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:22.080086 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:22.080058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:24.682919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:24.682883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:24.821067 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:24.821035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:26.083199 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:26.083168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:26.183265 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:26.183227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:26.835697 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:26.835665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:26.835970 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:26.835730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:26.835970 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:26.835853    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:26.835970 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:26.835886    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 14:23:26.835970 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:26.835901    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc8c84dc8-6lv4c: secret "image-registry-tls" not found
Jul 28 14:23:26.835970 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:26.835931    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls podName:904fca85-96bf-4856-b032-c57d0ab7cfbc nodeName:}" failed. No retries permitted until 2026-07-28 14:23:42.83591409 +0000 UTC m=+66.282763377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls") pod "dns-default-px49m" (UID: "904fca85-96bf-4856-b032-c57d0ab7cfbc") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:26.835970 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:26.835952    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls podName:233e6a1c-28cf-4e70-9714-efbadb1331d9 nodeName:}" failed. No retries permitted until 2026-07-28 14:23:42.835938908 +0000 UTC m=+66.282788194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls") pod "image-registry-dc8c84dc8-6lv4c" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9") : secret "image-registry-tls" not found
Jul 28 14:23:29.060131 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:29.060090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:29.124071 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:29.124034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:30.250802 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:30.250769    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:30.320858 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:30.320826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:36.445222 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:36.445195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xvvg4"
Jul 28 14:23:36.616001 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:36.615968    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:37.914589 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:37.914556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:41.749724 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.749682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:23:41.750122 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.749753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:41.756112 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.756087    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 14:23:41.756216 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.756089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 14:23:41.760238 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:41.760219    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 14:23:41.760294 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:41.760288    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs podName:ed53709b-b3b8-4ef7-a9c2-35a9789dac55 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:45.760272042 +0000 UTC m=+129.207121329 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs") pod "network-metrics-daemon-h5w88" (UID: "ed53709b-b3b8-4ef7-a9c2-35a9789dac55") : secret "metrics-daemon-secret" not found
Jul 28 14:23:41.763764 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.763746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a00f8f97-a454-45d2-9758-e09e9f0ff6f3-original-pull-secret\") pod \"global-pull-secret-syncer-txplh\" (UID: \"a00f8f97-a454-45d2-9758-e09e9f0ff6f3\") " pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:41.851138 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.851107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:41.853830 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.853812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 14:23:41.857928 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.857913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-txplh"
Jul 28 14:23:41.864533 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.864515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 14:23:41.874609 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:41.874589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6hfv\" (UniqueName: \"kubernetes.io/projected/6dfe72c3-55e9-4cee-916d-8e7e88301f67-kube-api-access-j6hfv\") pod \"network-check-target-92xq5\" (UID: \"6dfe72c3-55e9-4cee-916d-8e7e88301f67\") " pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:42.004985 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:42.004925    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-txplh"]
Jul 28 14:23:42.008137 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:23:42.008108    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00f8f97_a454_45d2_9758_e09e9f0ff6f3.slice/crio-37f9ab63526b76977bc255ec1c57c812b4c7139f0169c9976ee508fbbe2f6517 WatchSource:0}: Error finding container 37f9ab63526b76977bc255ec1c57c812b4c7139f0169c9976ee508fbbe2f6517: Status 404 returned error can't find the container with id 37f9ab63526b76977bc255ec1c57c812b4c7139f0169c9976ee508fbbe2f6517
Jul 28 14:23:42.169865 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:42.169837    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kq7zv\""
Jul 28 14:23:42.177767 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:42.177747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:42.315889 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:42.315855    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-92xq5"]
Jul 28 14:23:42.319493 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:23:42.319471    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfe72c3_55e9_4cee_916d_8e7e88301f67.slice/crio-daa3661d5ca0afd80ee7437eeb4c09e4a7cbe4bdbd2864e638eb5716d6fdddb1 WatchSource:0}: Error finding container daa3661d5ca0afd80ee7437eeb4c09e4a7cbe4bdbd2864e638eb5716d6fdddb1: Status 404 returned error can't find the container with id daa3661d5ca0afd80ee7437eeb4c09e4a7cbe4bdbd2864e638eb5716d6fdddb1
Jul 28 14:23:42.585943 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:42.585904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txplh" event={"ID":"a00f8f97-a454-45d2-9758-e09e9f0ff6f3","Type":"ContainerStarted","Data":"37f9ab63526b76977bc255ec1c57c812b4c7139f0169c9976ee508fbbe2f6517"}
Jul 28 14:23:42.586931 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:42.586909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-92xq5" event={"ID":"6dfe72c3-55e9-4cee-916d-8e7e88301f67","Type":"ContainerStarted","Data":"daa3661d5ca0afd80ee7437eeb4c09e4a7cbe4bdbd2864e638eb5716d6fdddb1"}
Jul 28 14:23:42.858695 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:42.858614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:23:42.859123 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:42.858727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:23:42.859123 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:42.858792    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 14:23:42.859123 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:42.858814    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc8c84dc8-6lv4c: secret "image-registry-tls" not found
Jul 28 14:23:42.859123 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:42.858867    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:23:42.859123 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:42.858885    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls podName:233e6a1c-28cf-4e70-9714-efbadb1331d9 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:14.858864966 +0000 UTC m=+98.305714262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls") pod "image-registry-dc8c84dc8-6lv4c" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9") : secret "image-registry-tls" not found
Jul 28 14:23:42.859123 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:23:42.858927    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls podName:904fca85-96bf-4856-b032-c57d0ab7cfbc nodeName:}" failed. No retries permitted until 2026-07-28 14:24:14.858908083 +0000 UTC m=+98.305757385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls") pod "dns-default-px49m" (UID: "904fca85-96bf-4856-b032-c57d0ab7cfbc") : secret "dns-default-metrics-tls" not found
Jul 28 14:23:43.212205 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:43.212131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:43.276922 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:43.276884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:45.599946 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:45.599857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-92xq5" event={"ID":"6dfe72c3-55e9-4cee-916d-8e7e88301f67","Type":"ContainerStarted","Data":"b63ebfd192ae7ac0dd9dc6d0ff407f99d1044ac1b0474ef0b9e5fbbfc666f9b7"}
Jul 28 14:23:45.600364 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:45.600005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:23:47.606948 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:47.606912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-txplh" event={"ID":"a00f8f97-a454-45d2-9758-e09e9f0ff6f3","Type":"ContainerStarted","Data":"d349619efddc0efd2529669599b03cd4d108b26991b3c1a6d94aa15e0a73f7fa"}
Jul 28 14:23:47.623409 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:47.623342    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-txplh" podStartSLOduration=65.788419444 podStartE2EDuration="1m10.623324101s" totalImagesPullingTime="4.834904657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:47.62307924 +0000 UTC m=+71.069928550" watchObservedRunningTime="2026-07-28 14:23:47.623324101 +0000 UTC m=+71.070173409"
Jul 28 14:23:47.623753 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:47.623725    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-92xq5" podStartSLOduration=67.641765543 podStartE2EDuration="1m10.623716745s" totalImagesPullingTime="2.981951202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:45.614881899 +0000 UTC m=+69.061731217" watchObservedRunningTime="2026-07-28 14:23:47.623716745 +0000 UTC m=+71.070566032"
Jul 28 14:23:51.018506 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:51.018467    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:23:51.119407 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:23:51.119353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:09.828743 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:09.828605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:09.895227 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:09.895199    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:11.194160 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.194126    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj"]
Jul 28 14:24:11.196887 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.196870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj"
Jul 28 14:24:11.198966 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.198946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rzmc7\""
Jul 28 14:24:11.207457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.207435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj"]
Jul 28 14:24:11.272718 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.272689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxjn\" (UniqueName: \"kubernetes.io/projected/457c21f0-f04a-403c-a8bd-b7a1672a0bb3-kube-api-access-mfxjn\") pod \"network-check-source-7d88cb55d8-kqvtj\" (UID: \"457c21f0-f04a-403c-a8bd-b7a1672a0bb3\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj"
Jul 28 14:24:11.373867 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.373836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxjn\" (UniqueName: \"kubernetes.io/projected/457c21f0-f04a-403c-a8bd-b7a1672a0bb3-kube-api-access-mfxjn\") pod \"network-check-source-7d88cb55d8-kqvtj\" (UID: \"457c21f0-f04a-403c-a8bd-b7a1672a0bb3\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj"
Jul 28 14:24:11.382674 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.382642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxjn\" (UniqueName: \"kubernetes.io/projected/457c21f0-f04a-403c-a8bd-b7a1672a0bb3-kube-api-access-mfxjn\") pod \"network-check-source-7d88cb55d8-kqvtj\" (UID: \"457c21f0-f04a-403c-a8bd-b7a1672a0bb3\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj"
Jul 28 14:24:11.506035 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.505947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj"
Jul 28 14:24:11.625938 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.625911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj"]
Jul 28 14:24:11.629329 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:24:11.629304    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457c21f0_f04a_403c_a8bd_b7a1672a0bb3.slice/crio-ffd4c1c4c2b009225580a59d37af6bbfba0ba0fb2dbaf1afa4d8fb2f0cb20c04 WatchSource:0}: Error finding container ffd4c1c4c2b009225580a59d37af6bbfba0ba0fb2dbaf1afa4d8fb2f0cb20c04: Status 404 returned error can't find the container with id ffd4c1c4c2b009225580a59d37af6bbfba0ba0fb2dbaf1afa4d8fb2f0cb20c04
Jul 28 14:24:11.675113 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:11.675081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj" event={"ID":"457c21f0-f04a-403c-a8bd-b7a1672a0bb3","Type":"ContainerStarted","Data":"ffd4c1c4c2b009225580a59d37af6bbfba0ba0fb2dbaf1afa4d8fb2f0cb20c04"}
Jul 28 14:24:12.447786 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:12.447747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:12.520043 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:12.520007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:12.679642 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:12.679608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj" event={"ID":"457c21f0-f04a-403c-a8bd-b7a1672a0bb3","Type":"ContainerStarted","Data":"69cc31a519f5cd1600e5df2528697e904b308deac68daa8a1ded237df262a316"}
Jul 28 14:24:12.695536 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:12.695462    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kqvtj" podStartSLOduration=1.695447815 podStartE2EDuration="1.695447815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:12.694930257 +0000 UTC m=+96.141779568" watchObservedRunningTime="2026-07-28 14:24:12.695447815 +0000 UTC m=+96.142297124"
Jul 28 14:24:14.293062 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.293023    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:14.385635 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.385611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:14.900986 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.900953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:24:14.901171 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.901000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") pod \"image-registry-dc8c84dc8-6lv4c\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") " pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:24:14.901171 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:14.901116    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 14:24:14.901171 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:14.901138    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 14:24:14.901171 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:14.901150    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dc8c84dc8-6lv4c: secret "image-registry-tls" not found
Jul 28 14:24:14.901354 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:14.901201    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls podName:233e6a1c-28cf-4e70-9714-efbadb1331d9 nodeName:}" failed. No retries permitted until 2026-07-28 14:25:18.90118108 +0000 UTC m=+162.348030373 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls") pod "image-registry-dc8c84dc8-6lv4c" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9") : secret "image-registry-tls" not found
Jul 28 14:24:14.901354 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:14.901217    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls podName:904fca85-96bf-4856-b032-c57d0ab7cfbc nodeName:}" failed. No retries permitted until 2026-07-28 14:25:18.901209291 +0000 UTC m=+162.348058579 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls") pod "dns-default-px49m" (UID: "904fca85-96bf-4856-b032-c57d0ab7cfbc") : secret "dns-default-metrics-tls" not found
Jul 28 14:24:14.985256 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.985226    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kmzdr"]
Jul 28 14:24:14.993986 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.993960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:14.996327 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.996307    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 14:24:14.996579 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.996557    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wl4mt\""
Jul 28 14:24:14.996695 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.996636    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 14:24:14.996695 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.996678    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 14:24:14.997001 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:14.996987    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 14:24:15.002351 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.002329    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmzdr"]
Jul 28 14:24:15.103462 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.103430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d84cfec-3afe-40e2-a59a-1a35e6b09523-crio-socket\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.103601 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.103486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.103601 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.103555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d84cfec-3afe-40e2-a59a-1a35e6b09523-data-volume\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.103701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.103620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.103701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.103651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pn4\" (UniqueName: \"kubernetes.io/projected/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-api-access-j4pn4\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.111965 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.111938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:15.173149 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.173084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:15.204286 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.204256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.204386 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.204302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d84cfec-3afe-40e2-a59a-1a35e6b09523-data-volume\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.204386 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.204346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.204386 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.204371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pn4\" (UniqueName: \"kubernetes.io/projected/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-api-access-j4pn4\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.204386 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:15.204378    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:15.204552 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:15.204468    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:15.704447903 +0000 UTC m=+99.151297208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:15.204552 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.204502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d84cfec-3afe-40e2-a59a-1a35e6b09523-crio-socket\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.204552 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:15.204531    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:15.204696 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:15.204593    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:15.704575699 +0000 UTC m=+99.151425003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:15.204696 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.204601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d84cfec-3afe-40e2-a59a-1a35e6b09523-crio-socket\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.204802 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.204731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d84cfec-3afe-40e2-a59a-1a35e6b09523-data-volume\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.213063 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.213037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pn4\" (UniqueName: \"kubernetes.io/projected/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-api-access-j4pn4\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.356412 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.356371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:15.495951 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.495864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:15.708764 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.708728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.708954 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.708800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:15.708954 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:15.708880    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:15.708954 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:15.708889    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:15.708954 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:15.708936    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:16.708922339 +0000 UTC m=+100.155771625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:15.708954 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:15.708948    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:16.708942599 +0000 UTC m=+100.155791887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:15.712299 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.712276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:15.812505 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:15.812451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:16.143121 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:16.143022    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:16.209772 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:16.209739    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:16.455447 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:16.455345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:16.554967 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:16.554930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:16.605348 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:16.605318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-92xq5"
Jul 28 14:24:16.717114 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:16.717022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:16.717287 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:16.717118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:16.717287 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:16.717189    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:16.717287 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:16.717200    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:16.717287 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:16.717255    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:18.717234721 +0000 UTC m=+102.164084013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:16.717287 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:16.717271    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:18.71726317 +0000 UTC m=+102.164112464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:17.171557 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:17.171529    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nxbfp_5edec3b9-e837-4bb1-9ec5-ffe2163f36df/dns-node-resolver/0.log"
Jul 28 14:24:17.327441 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:17.327405    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:17.428874 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:17.428771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:18.057267 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:18.057225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:18.124007 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:18.123978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:18.172067 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:18.172044    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wrgjj_6921424a-8439-4602-acc4-e525c0779616/node-ca/0.log"
Jul 28 14:24:18.735511 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:18.735470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:18.735699 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:18.735534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:18.735699 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:18.735588    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:18.735699 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:18.735653    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:22.735639362 +0000 UTC m=+106.182488649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:18.735699 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:18.735656    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:18.735699 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:18.735701    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:22.735688365 +0000 UTC m=+106.182537652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:18.774812 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:18.774777    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kjgps_5979f987-8fc4-42fc-9729-6db64e0c42bd/serve-healthcheck-canary/0.log"
Jul 28 14:24:19.168590 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:19.168559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:21.766357 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:21.766326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:21.867120 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:21.867080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:21.947220 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:21.947189    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:22.126307 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:22.126273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:22.727399 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:22.727358    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:22.766436 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:22.766411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:22.766725 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:22.766531    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:22.766725 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:22.766545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:22.766725 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:22.766616    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 14:24:22.766725 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:22.766619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:30.766604908 +0000 UTC m=+114.213454195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:22.766725 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:22.766675    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:30.76666295 +0000 UTC m=+114.213512237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : configmap "kube-rbac-proxy" not found
Jul 28 14:24:22.830532 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:22.830497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:23.268682 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:23.268651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:24.184643 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:24.184611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:24.285711 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:24.285683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:24.363283 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:24.363260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:24.658185 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:24.658155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:25.073529 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:25.073500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:25.656408 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:25.656354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:25.726890 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:25.726861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:25.793471 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:25.793443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:26.262602 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:26.262571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:26.471205 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:26.471173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:26.538224 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:26.538162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:26.869929 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:26.869904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:27.057958 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:27.057918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:27.460820 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:27.460775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:27.535667 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:27.535640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:28.058736 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:28.058694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:28.289763 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:28.289730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:28.665403 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:28.665357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:28.857049 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:28.857019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:29.056407 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:29.056356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:29.459053 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:29.458979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:30.056414 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:30.056369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:30.457090 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:30.456996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:30.825344 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:30.825313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:30.825706 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:30.825420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:30.825706 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:30.825489    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:30.825706 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:24:30.825560    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls podName:9d84cfec-3afe-40e2-a59a-1a35e6b09523 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:46.825540004 +0000 UTC m=+130.272389301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmzdr" (UID: "9d84cfec-3afe-40e2-a59a-1a35e6b09523") : secret "insights-runtime-extractor-tls" not found
Jul 28 14:24:30.826068 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:30.826047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d84cfec-3afe-40e2-a59a-1a35e6b09523-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:30.861294 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:30.861261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:31.278647 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:31.278583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:31.344154 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:31.344123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:31.458694 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:31.458665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:31.867434 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:31.867379    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:33.091880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:33.091842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:33.151740 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:33.151704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:39.722190 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:39.722152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:39.818030 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:39.818003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:39.917960 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:39.917929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:41.088381 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:41.088337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:41.155639 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:41.155602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:41.341611 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:41.341542    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" podUID="de66e80c-b431-485e-bc0b-ea3eede95805" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 14:24:44.591312 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:44.591272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:44.692276 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:44.692248    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:44.782978 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:44.782935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:45.013940 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:45.013861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:45.836951 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:45.836923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:24:45.839455 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:45.839428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed53709b-b3b8-4ef7-a9c2-35a9789dac55-metrics-certs\") pod \"network-metrics-daemon-h5w88\" (UID: \"ed53709b-b3b8-4ef7-a9c2-35a9789dac55\") " pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:24:46.065712 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:46.065687    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z6ktw\""
Jul 28 14:24:46.073842 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:46.073818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5w88"
Jul 28 14:24:46.197690 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:46.197665    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h5w88"]
Jul 28 14:24:46.201460 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:24:46.201430    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded53709b_b3b8_4ef7_a9c2_35a9789dac55.slice/crio-f8eb5dda1efd805fe10c240f9c426a80c7eb72ed733a4879ef1a7da350a7d095 WatchSource:0}: Error finding container f8eb5dda1efd805fe10c240f9c426a80c7eb72ed733a4879ef1a7da350a7d095: Status 404 returned error can't find the container with id f8eb5dda1efd805fe10c240f9c426a80c7eb72ed733a4879ef1a7da350a7d095
Jul 28 14:24:46.786504 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:46.786471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5w88" event={"ID":"ed53709b-b3b8-4ef7-a9c2-35a9789dac55","Type":"ContainerStarted","Data":"f8eb5dda1efd805fe10c240f9c426a80c7eb72ed733a4879ef1a7da350a7d095"}
Jul 28 14:24:46.844131 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:46.844084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:46.846981 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:46.846955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d84cfec-3afe-40e2-a59a-1a35e6b09523-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmzdr\" (UID: \"9d84cfec-3afe-40e2-a59a-1a35e6b09523\") " pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:47.105979 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.105950    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wl4mt\""
Jul 28 14:24:47.113810 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.113785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmzdr"
Jul 28 14:24:47.262858 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.262832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmzdr"]
Jul 28 14:24:47.265731 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:24:47.265704    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d84cfec_3afe_40e2_a59a_1a35e6b09523.slice/crio-f79d8014d36d3d76f74614a87a76ab6574ca491d24c642e698c252634e9d1180 WatchSource:0}: Error finding container f79d8014d36d3d76f74614a87a76ab6574ca491d24c642e698c252634e9d1180: Status 404 returned error can't find the container with id f79d8014d36d3d76f74614a87a76ab6574ca491d24c642e698c252634e9d1180
Jul 28 14:24:47.793506 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.793466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmzdr" event={"ID":"9d84cfec-3afe-40e2-a59a-1a35e6b09523","Type":"ContainerStarted","Data":"f79d8014d36d3d76f74614a87a76ab6574ca491d24c642e698c252634e9d1180"}
Jul 28 14:24:47.795769 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.795740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:47.796308 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.796282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5w88" event={"ID":"ed53709b-b3b8-4ef7-a9c2-35a9789dac55","Type":"ContainerStarted","Data":"f8c12de1e9c44969b4f31f9bbb7b616e27c8d790d59d5db951a3e2e4c1452d98"}
Jul 28 14:24:47.796425 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.796328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5w88" event={"ID":"ed53709b-b3b8-4ef7-a9c2-35a9789dac55","Type":"ContainerStarted","Data":"44abbd2464a27bbe9b4617fbb9c9805541a63febc9c8cf138a1ae519caa86a94"}
Jul 28 14:24:47.818520 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.818473    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h5w88" podStartSLOduration=129.864868852 podStartE2EDuration="2m10.818454849s" totalImagesPullingTime="953.585997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:47.817441525 +0000 UTC m=+131.264290836" watchObservedRunningTime="2026-07-28 14:24:47.818454849 +0000 UTC m=+131.265304164"
Jul 28 14:24:47.872971 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.872945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:47.946072 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:47.946027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:48.046491 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:48.046430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:48.254973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:48.254930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:51.253366 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:51.253331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:51.317220 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:51.317191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:51.341554 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:51.341525    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" podUID="de66e80c-b431-485e-bc0b-ea3eede95805" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 14:24:51.488534 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:51.488501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:51.549218 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:51.549182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:52.821672 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:52.821632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmzdr" event={"ID":"9d84cfec-3afe-40e2-a59a-1a35e6b09523","Type":"ContainerStarted","Data":"d7faec488da1f3693959ee489a021afeb9d59b46ec8c0c0734eddd3faca7df5b"}
Jul 28 14:24:52.866552 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:52.866503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:54.978904 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:54.978865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:55.075273 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.075211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:55.189664 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.189628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:55.260075 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.260004    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lftxv"]
Jul 28 14:24:55.262218 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.262194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.269318 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.269293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 14:24:55.269697 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.269678    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5brqd\""
Jul 28 14:24:55.270192 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.270174    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 14:24:55.270265 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.270235    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 14:24:55.270760 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.270741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 14:24:55.270855 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.270818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 14:24:55.271725 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.271708    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 14:24:55.306943 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.306916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-root\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.307066 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.306957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.307122 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.307076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-textfile\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.307162 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.307132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpzf\" (UniqueName: \"kubernetes.io/projected/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-kube-api-access-qfpzf\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.307242 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.307221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-wtmp\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.307321 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.307304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-metrics-client-ca\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.307370 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.307339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-tls\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.307505 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.307461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-sys\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.307572 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.307527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.324923 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.324897    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:55.408376 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-textfile\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408452 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpzf\" (UniqueName: \"kubernetes.io/projected/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-kube-api-access-qfpzf\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408506 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-wtmp\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408506 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-metrics-client-ca\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408506 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-tls\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408652 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-sys\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408652 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408652 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-root\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408652 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408652 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-wtmp\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408884 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-textfile\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408884 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-sys\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.408884 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.408752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-root\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.409030 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.409012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-metrics-client-ca\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.409140 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.409115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.410597 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.410576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-tls\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.411071 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.411053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.418242 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.418205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpzf\" (UniqueName: \"kubernetes.io/projected/6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1-kube-api-access-qfpzf\") pod \"node-exporter-lftxv\" (UID: \"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1\") " pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.430004 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.429981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:55.521386 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.521335    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:55.571971 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.571948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lftxv"
Jul 28 14:24:55.580159 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:24:55.580136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5a8157_ece4_4714_a9c6_bb95c8b2b5e1.slice/crio-9eaf9553473d0be7e47b11e483e1f7f0c3be51b3281a170301bf7ebe3af53f57 WatchSource:0}: Error finding container 9eaf9553473d0be7e47b11e483e1f7f0c3be51b3281a170301bf7ebe3af53f57: Status 404 returned error can't find the container with id 9eaf9553473d0be7e47b11e483e1f7f0c3be51b3281a170301bf7ebe3af53f57
Jul 28 14:24:55.616963 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.616935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:55.802677 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.802623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:55.835543 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.835504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmzdr" event={"ID":"9d84cfec-3afe-40e2-a59a-1a35e6b09523","Type":"ContainerStarted","Data":"bce0b302e45687e480f23db35e861af10a6a0ca39a34cfc5714eab0e750c0175"}
Jul 28 14:24:55.837080 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.837058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lftxv" event={"ID":"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1","Type":"ContainerStarted","Data":"9eaf9553473d0be7e47b11e483e1f7f0c3be51b3281a170301bf7ebe3af53f57"}
Jul 28 14:24:55.903093 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:55.903069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:56.235675 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:56.235595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:56.426197 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:56.426156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:56.507448 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:56.507357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:56.841687 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:56.841658    2579 generic.go:356] "Generic (PLEG): container finished" podID="6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1" containerID="33d3ef35ffa7fd92b46cf932d9f83a0e03dfc7524f4de0fd729136556937ad4e" exitCode=0
Jul 28 14:24:56.841809 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:56.841725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lftxv" event={"ID":"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1","Type":"ContainerDied","Data":"33d3ef35ffa7fd92b46cf932d9f83a0e03dfc7524f4de0fd729136556937ad4e"}
Jul 28 14:24:57.848142 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:57.848103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lftxv" event={"ID":"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1","Type":"ContainerStarted","Data":"300687657d68b1e76f8caea28f9959bd382c904c6fb77f560b820e63f440c50f"}
Jul 28 14:24:57.848558 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:57.848165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lftxv" event={"ID":"6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1","Type":"ContainerStarted","Data":"de18e58e7a13ae6c92f7af0571056089c56070b5429e301368a3408d4e83b899"}
Jul 28 14:24:57.874304 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:57.874252    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lftxv" podStartSLOduration=1.721001754 podStartE2EDuration="2.874231457s" totalImagesPullingTime="1.079595114s" totalInitContainerRuntime="73.634589ms" isStatefulPod=true podCreationTimestamp="2026-07-28 14:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:57.873411544 +0000 UTC m=+141.320260849" watchObservedRunningTime="2026-07-28 14:24:57.874231457 +0000 UTC m=+141.321080766"
Jul 28 14:24:58.338686 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:58.338649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:58.430949 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:58.430911    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:58.746359 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:58.746279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:59.028432 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:59.028358    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:24:59.857757 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:59.857716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmzdr" event={"ID":"9d84cfec-3afe-40e2-a59a-1a35e6b09523","Type":"ContainerStarted","Data":"6548388ca17170b141061f07932be62ae61db9e806c03c6f326b0d7b221010bc"}
Jul 28 14:24:59.877000 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:59.876949    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kmzdr" podStartSLOduration=34.377913463 podStartE2EDuration="45.876931332s" totalImagesPullingTime="11.499017869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:24:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:59.875943256 +0000 UTC m=+143.322792567" watchObservedRunningTime="2026-07-28 14:24:59.876931332 +0000 UTC m=+143.323780701"
Jul 28 14:24:59.971720 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:24:59.971677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:00.171793 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:00.171732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:00.552576 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:00.552552    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:01.189173 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.189138    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dc8c84dc8-6lv4c"]
Jul 28 14:25:01.189531 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:25:01.189350    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c" podUID="233e6a1c-28cf-4e70-9714-efbadb1331d9"
Jul 28 14:25:01.214230 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.214202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:01.315261 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.315240    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:01.340744 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.340717    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" podUID="de66e80c-b431-485e-bc0b-ea3eede95805" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 14:25:01.340863 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.340788    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr"
Jul 28 14:25:01.341247 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.341212    2579 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"d61316212debd0375df13d5dd170ef1fd57da6bc788ffbc70853ca83829d7d2e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 28 14:25:01.341352 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.341272    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" podUID="de66e80c-b431-485e-bc0b-ea3eede95805" containerName="service-proxy" containerID="cri-o://d61316212debd0375df13d5dd170ef1fd57da6bc788ffbc70853ca83829d7d2e" gracePeriod=30
Jul 28 14:25:01.419877 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.419851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:01.687026 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.686996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:01.866431 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.866374    2579 generic.go:356] "Generic (PLEG): container finished" podID="de66e80c-b431-485e-bc0b-ea3eede95805" containerID="d61316212debd0375df13d5dd170ef1fd57da6bc788ffbc70853ca83829d7d2e" exitCode=2
Jul 28 14:25:01.866565 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.866462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" event={"ID":"de66e80c-b431-485e-bc0b-ea3eede95805","Type":"ContainerDied","Data":"d61316212debd0375df13d5dd170ef1fd57da6bc788ffbc70853ca83829d7d2e"}
Jul 28 14:25:01.866565 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.866516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b9457d8b5-w6tsr" event={"ID":"de66e80c-b431-485e-bc0b-ea3eede95805","Type":"ContainerStarted","Data":"69c1c3e59ac41ab0e96f1ecf927f47c4fefc1f39e0bb1f80c1b244332adf4d60"}
Jul 28 14:25:01.866565 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.866475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:25:01.873244 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.873214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:25:01.957312 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-bound-sa-token\") pod \"233e6a1c-28cf-4e70-9714-efbadb1331d9\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") "
Jul 28 14:25:01.957445 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957324    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-installation-pull-secrets\") pod \"233e6a1c-28cf-4e70-9714-efbadb1331d9\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") "
Jul 28 14:25:01.957445 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957343    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-image-registry-private-configuration\") pod \"233e6a1c-28cf-4e70-9714-efbadb1331d9\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") "
Jul 28 14:25:01.957445 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957361    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-certificates\") pod \"233e6a1c-28cf-4e70-9714-efbadb1331d9\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") "
Jul 28 14:25:01.957445 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mv8m\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-kube-api-access-8mv8m\") pod \"233e6a1c-28cf-4e70-9714-efbadb1331d9\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") "
Jul 28 14:25:01.957654 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-trusted-ca\") pod \"233e6a1c-28cf-4e70-9714-efbadb1331d9\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") "
Jul 28 14:25:01.957654 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957577    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/233e6a1c-28cf-4e70-9714-efbadb1331d9-ca-trust-extracted\") pod \"233e6a1c-28cf-4e70-9714-efbadb1331d9\" (UID: \"233e6a1c-28cf-4e70-9714-efbadb1331d9\") "
Jul 28 14:25:01.958209 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957793    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "233e6a1c-28cf-4e70-9714-efbadb1331d9" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:25:01.958209 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957926    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-certificates\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:25:01.958209 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.957986    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "233e6a1c-28cf-4e70-9714-efbadb1331d9" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:25:01.958209 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.958156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/233e6a1c-28cf-4e70-9714-efbadb1331d9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "233e6a1c-28cf-4e70-9714-efbadb1331d9" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:25:01.959848 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.959822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "233e6a1c-28cf-4e70-9714-efbadb1331d9" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:25:01.959963 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.959924    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "233e6a1c-28cf-4e70-9714-efbadb1331d9" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:25:01.960014 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.959954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "233e6a1c-28cf-4e70-9714-efbadb1331d9" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:25:01.960053 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:01.960031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-kube-api-access-8mv8m" (OuterVolumeSpecName: "kube-api-access-8mv8m") pod "233e6a1c-28cf-4e70-9714-efbadb1331d9" (UID: "233e6a1c-28cf-4e70-9714-efbadb1331d9"). InnerVolumeSpecName "kube-api-access-8mv8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:25:02.058714 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.058689    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/233e6a1c-28cf-4e70-9714-efbadb1331d9-trusted-ca\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:25:02.058714 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.058709    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/233e6a1c-28cf-4e70-9714-efbadb1331d9-ca-trust-extracted\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:25:02.058838 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.058719    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-bound-sa-token\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:25:02.058838 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.058728    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-installation-pull-secrets\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:25:02.058838 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.058737    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/233e6a1c-28cf-4e70-9714-efbadb1331d9-image-registry-private-configuration\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:25:02.058838 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.058745    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mv8m\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-kube-api-access-8mv8m\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:25:02.869940 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.869908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dc8c84dc8-6lv4c"
Jul 28 14:25:02.904638 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.904608    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dc8c84dc8-6lv4c"]
Jul 28 14:25:02.908243 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.908217    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-dc8c84dc8-6lv4c"]
Jul 28 14:25:02.964715 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:02.964688    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/233e6a1c-28cf-4e70-9714-efbadb1331d9-registry-tls\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:25:03.250296 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:03.250221    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233e6a1c-28cf-4e70-9714-efbadb1331d9" path="/var/lib/kubelet/pods/233e6a1c-28cf-4e70-9714-efbadb1331d9/volumes"
Jul 28 14:25:03.465009 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:03.464980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:04.347410 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:04.347367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:04.448588 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:04.448558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:05.474679 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:05.474626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:05.587036 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:05.586999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:10.615185 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:10.615152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:10.715219 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:10.715187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:10.815677 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:10.815650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:13.164641 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:13.164594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:13.267288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:13.267253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:13.578928 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:13.578900    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:14.068766 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:25:14.068735    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-px49m" podUID="904fca85-96bf-4856-b032-c57d0ab7cfbc"
Jul 28 14:25:14.907261 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:14.907233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-px49m"
Jul 28 14:25:16.591121 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:16.591092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:16.691539 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:16.691510    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:18.975736 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:18.975692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:25:18.977934 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:18.977912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/904fca85-96bf-4856-b032-c57d0ab7cfbc-metrics-tls\") pod \"dns-default-px49m\" (UID: \"904fca85-96bf-4856-b032-c57d0ab7cfbc\") " pod="openshift-dns/dns-default-px49m"
Jul 28 14:25:18.991922 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:18.991890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:19.066676 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:19.066653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:19.110493 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:19.110470    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pl2dt\""
Jul 28 14:25:19.118529 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:19.118514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-px49m"
Jul 28 14:25:19.249560 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:25:19.249534    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904fca85_96bf_4856_b032_c57d0ab7cfbc.slice/crio-fc1df58631228d742b508fcaa3b112fa1ee2b5972713b027bc73d85fc63eeb7d WatchSource:0}: Error finding container fc1df58631228d742b508fcaa3b112fa1ee2b5972713b027bc73d85fc63eeb7d: Status 404 returned error can't find the container with id fc1df58631228d742b508fcaa3b112fa1ee2b5972713b027bc73d85fc63eeb7d
Jul 28 14:25:19.250286 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:19.250193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-px49m"]
Jul 28 14:25:19.924239 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:19.924205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-px49m" event={"ID":"904fca85-96bf-4856-b032-c57d0ab7cfbc","Type":"ContainerStarted","Data":"fc1df58631228d742b508fcaa3b112fa1ee2b5972713b027bc73d85fc63eeb7d"}
Jul 28 14:25:21.933612 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:21.933578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-px49m" event={"ID":"904fca85-96bf-4856-b032-c57d0ab7cfbc","Type":"ContainerStarted","Data":"0d4e66e1c86e942125fd16f207f20ead1cda198920da4cb72b761a2b0c8c93c3"}
Jul 28 14:25:21.933995 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:21.933645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-px49m" event={"ID":"904fca85-96bf-4856-b032-c57d0ab7cfbc","Type":"ContainerStarted","Data":"2dc8dc029c50f7dfa82eb83ce63cb22a0f5101eb6dfdc3fe7b7f459db136daac"}
Jul 28 14:25:21.933995 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:21.933705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-px49m"
Jul 28 14:25:21.952880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:21.952836    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-px49m" podStartSLOduration=130.198086432 podStartE2EDuration="2m11.95282162s" totalImagesPullingTime="1.754735188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:21.951734483 +0000 UTC m=+165.398583818" watchObservedRunningTime="2026-07-28 14:25:21.95282162 +0000 UTC m=+165.399670929"
Jul 28 14:25:25.691103 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:25.691063    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:26.495978 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:26.495933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:26.569033 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:26.568992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:29.588372 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:29.588274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:29.685678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:29.685635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:31.939349 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:31.939312    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-px49m"
Jul 28 14:25:38.821045 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:38.821004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:38.885702 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:38.885670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:40.999452 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:40.999413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:41.291135 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:41.291047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:41.397263 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:41.396371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:42.387149 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:42.387107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:42.487724 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:42.487690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:43.785845 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:43.785799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:50.093880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:50.093834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:50.159112 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:50.159080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:51.355266 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:51.355230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:56.072751 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:56.072709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:25:56.143198 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:25:56.143162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:26:14.567741 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:26:14.567703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:26:14.960309 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:26:14.960221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:26:23.548865 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:26:23.548823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:26:23.648806 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:26:23.648778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:26:30.926973 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:26:30.926930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:26:34.922515 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:26:34.922474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:26:35.019723 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:26:35.019691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:26:35.099372 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:26:35.099345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:01.222868 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:01.222777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:15.620120 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:15.620075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:16.209521 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:16.209483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:29.843298 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:29.843255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:29.892269 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:29.892236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:36.959545 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:36.959510    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 14:27:41.847160 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:41.847112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:41.914663 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:41.914625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:56.075444 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:56.075381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:27:56.179043 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:27:56.178993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:28:02.762633 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:28:02.762593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:28:02.863999 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:28:02.863963    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:28:13.267779 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:28:13.267743    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:28:13.324969 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:28:13.324928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:28:13.378462 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:28:13.378436    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:28:57.263950 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:28:57.263919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:28:59.793875 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:28:59.793832    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:18.897576 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:18.897540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:19.728122 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:19.728087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:22.539700 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:22.539661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:22.640331 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:22.640293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:24.898117 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:24.898088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:25.030983 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:25.030954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:27.161159 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:27.161125    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:27.636922 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:27.636885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:28.353784 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.353757    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"]
Jul 28 14:29:28.356703 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.356685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"
Jul 28 14:29:28.359920 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.359893    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 14:29:28.360031 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.359944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-c9tsl\""
Jul 28 14:29:28.360691 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.360679    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 14:29:28.371658 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.371626    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"]
Jul 28 14:29:28.442877 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.442848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrkz\" (UniqueName: \"kubernetes.io/projected/c2ad959a-8569-47dd-a491-bfd9ed7c1cb0-kube-api-access-dcrkz\") pod \"cert-manager-cainjector-559cdd8c87-xbm55\" (UID: \"c2ad959a-8569-47dd-a491-bfd9ed7c1cb0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"
Jul 28 14:29:28.442980 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.442912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ad959a-8569-47dd-a491-bfd9ed7c1cb0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xbm55\" (UID: \"c2ad959a-8569-47dd-a491-bfd9ed7c1cb0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"
Jul 28 14:29:28.544109 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.544085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrkz\" (UniqueName: \"kubernetes.io/projected/c2ad959a-8569-47dd-a491-bfd9ed7c1cb0-kube-api-access-dcrkz\") pod \"cert-manager-cainjector-559cdd8c87-xbm55\" (UID: \"c2ad959a-8569-47dd-a491-bfd9ed7c1cb0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"
Jul 28 14:29:28.544215 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.544128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ad959a-8569-47dd-a491-bfd9ed7c1cb0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xbm55\" (UID: \"c2ad959a-8569-47dd-a491-bfd9ed7c1cb0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"
Jul 28 14:29:28.551397 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.551378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrkz\" (UniqueName: \"kubernetes.io/projected/c2ad959a-8569-47dd-a491-bfd9ed7c1cb0-kube-api-access-dcrkz\") pod \"cert-manager-cainjector-559cdd8c87-xbm55\" (UID: \"c2ad959a-8569-47dd-a491-bfd9ed7c1cb0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"
Jul 28 14:29:28.551950 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.551930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2ad959a-8569-47dd-a491-bfd9ed7c1cb0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xbm55\" (UID: \"c2ad959a-8569-47dd-a491-bfd9ed7c1cb0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"
Jul 28 14:29:28.664774 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.664722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"
Jul 28 14:29:28.781147 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.781116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xbm55"]
Jul 28 14:29:28.783788 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:29:28.783749    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ad959a_8569_47dd_a491_bfd9ed7c1cb0.slice/crio-3e8e099feb758b95e81d00275a8f27f9bdbfdabd26bafb06a2d4f9573c19c1c7 WatchSource:0}: Error finding container 3e8e099feb758b95e81d00275a8f27f9bdbfdabd26bafb06a2d4f9573c19c1c7: Status 404 returned error can't find the container with id 3e8e099feb758b95e81d00275a8f27f9bdbfdabd26bafb06a2d4f9573c19c1c7
Jul 28 14:29:28.785619 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:28.785605    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:29:29.313961 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:29.313933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:29.368858 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:29.368807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:29.716943 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:29.716859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55" event={"ID":"c2ad959a-8569-47dd-a491-bfd9ed7c1cb0","Type":"ContainerStarted","Data":"3e8e099feb758b95e81d00275a8f27f9bdbfdabd26bafb06a2d4f9573c19c1c7"}
Jul 28 14:29:31.725160 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:31.725135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55" event={"ID":"c2ad959a-8569-47dd-a491-bfd9ed7c1cb0","Type":"ContainerStarted","Data":"e5a405cfaf2fa7fbdc606082ab5cb93f098ef1bc6bd664464300e4f58f69eb2e"}
Jul 28 14:29:31.765037 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:31.764985    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbm55" podStartSLOduration=0.988658986 podStartE2EDuration="3.764964727s" totalImagesPullingTime="2.776305741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:29:31.764622979 +0000 UTC m=+415.211472288" watchObservedRunningTime="2026-07-28 14:29:31.764964727 +0000 UTC m=+415.211814041"
Jul 28 14:29:32.348921 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:32.348887    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:32.433532 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:32.433491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:34.388057 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:34.388021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:40.915484 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:40.915437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:41.010846 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:41.010782    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:43.728739 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:43.728689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:46.286567 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:46.286524    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:46.386923 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:46.386888    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:48.346656 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:48.346616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:56.561409 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:56.561344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:29:56.676855 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:29:56.676824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:02.718351 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:02.718252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:02.820779 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:02.820749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:03.328233 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.328191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:03.670322 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.670258    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"]
Jul 28 14:30:03.673375 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.673355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.675840 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.675818    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 14:30:03.675920 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.675836    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 14:30:03.676454 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.676426    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 14:30:03.676550 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.676432    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 14:30:03.677199 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.677174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vfcv9\""
Jul 28 14:30:03.689198 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.689177    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"]
Jul 28 14:30:03.771921 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.771892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ba14e51-17c9-4e46-937a-a63acea89d7e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.772206 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.771935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ba14e51-17c9-4e46-937a-a63acea89d7e-manifests\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.772206 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.771963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59wj\" (UniqueName: \"kubernetes.io/projected/3ba14e51-17c9-4e46-937a-a63acea89d7e-kube-api-access-r59wj\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.772206 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.772022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba14e51-17c9-4e46-937a-a63acea89d7e-tmp\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.772206 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.772047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ba14e51-17c9-4e46-937a-a63acea89d7e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.873300 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.873276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ba14e51-17c9-4e46-937a-a63acea89d7e-manifests\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.873434 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.873305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59wj\" (UniqueName: \"kubernetes.io/projected/3ba14e51-17c9-4e46-937a-a63acea89d7e-kube-api-access-r59wj\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.873434 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.873330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba14e51-17c9-4e46-937a-a63acea89d7e-tmp\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.873434 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.873347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ba14e51-17c9-4e46-937a-a63acea89d7e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.873609 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.873540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ba14e51-17c9-4e46-937a-a63acea89d7e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.873846 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.873824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ba14e51-17c9-4e46-937a-a63acea89d7e-manifests\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.875646 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.875614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ba14e51-17c9-4e46-937a-a63acea89d7e-tmp\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.875860 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.875843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ba14e51-17c9-4e46-937a-a63acea89d7e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.875988 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.875967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ba14e51-17c9-4e46-937a-a63acea89d7e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.880994 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.880973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59wj\" (UniqueName: \"kubernetes.io/projected/3ba14e51-17c9-4e46-937a-a63acea89d7e-kube-api-access-r59wj\") pod \"opendatahub-operator-controller-manager-6f457fb848-fzzdw\" (UID: \"3ba14e51-17c9-4e46-937a-a63acea89d7e\") " pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:03.988098 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:03.988045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:04.111919 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:04.111894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"]
Jul 28 14:30:04.115031 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:30:04.115001    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba14e51_17c9_4e46_937a_a63acea89d7e.slice/crio-7561790f0e0c28f176a6ccb2061e66b730a1c2aeb6247b4ed010c280bfa782ff WatchSource:0}: Error finding container 7561790f0e0c28f176a6ccb2061e66b730a1c2aeb6247b4ed010c280bfa782ff: Status 404 returned error can't find the container with id 7561790f0e0c28f176a6ccb2061e66b730a1c2aeb6247b4ed010c280bfa782ff
Jul 28 14:30:04.326032 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:04.326004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:04.421654 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:04.421631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:04.838364 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:04.838335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw" event={"ID":"3ba14e51-17c9-4e46-937a-a63acea89d7e","Type":"ContainerStarted","Data":"7561790f0e0c28f176a6ccb2061e66b730a1c2aeb6247b4ed010c280bfa782ff"}
Jul 28 14:30:07.856709 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:07.856676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw" event={"ID":"3ba14e51-17c9-4e46-937a-a63acea89d7e","Type":"ContainerStarted","Data":"08d088ab5c1b8066c52a2dc9c2a00c3053a1285e880ee6725149103978ac9aba"}
Jul 28 14:30:08.862520 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:08.862487    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ba14e51-17c9-4e46-937a-a63acea89d7e" containerID="08d088ab5c1b8066c52a2dc9c2a00c3053a1285e880ee6725149103978ac9aba" exitCode=0
Jul 28 14:30:08.862901 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:08.862588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw" event={"ID":"3ba14e51-17c9-4e46-937a-a63acea89d7e","Type":"ContainerDied","Data":"08d088ab5c1b8066c52a2dc9c2a00c3053a1285e880ee6725149103978ac9aba"}
Jul 28 14:30:09.868501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:09.868466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw" event={"ID":"3ba14e51-17c9-4e46-937a-a63acea89d7e","Type":"ContainerStarted","Data":"b0fb9284ad8310e672a22050497761d3649405625477b80e972cfdac26315674"}
Jul 28 14:30:09.868970 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:09.868593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:09.891897 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:09.891844    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw" podStartSLOduration=2.167822799 podStartE2EDuration="6.891820052s" totalImagesPullingTime="3.813630641s" totalInitContainerRuntime="910.366612ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:09.890773562 +0000 UTC m=+453.337622869" watchObservedRunningTime="2026-07-28 14:30:09.891820052 +0000 UTC m=+453.338669361"
Jul 28 14:30:20.874533 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:20.874494    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f457fb848-fzzdw"
Jul 28 14:30:24.383179 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.383147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:24.477075 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.477042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:24.706315 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.706226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:24.832668 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.832636    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"]
Jul 28 14:30:24.842881 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.842858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:24.845843 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.845815    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 14:30:24.845972 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.845879    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 14:30:24.846845 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.846821    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 14:30:24.846845 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.846821    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c9pfb\""
Jul 28 14:30:24.847081 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.847066    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 14:30:24.847573 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.847547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"]
Jul 28 14:30:24.910593 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.910568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-tls-certs\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:24.910701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.910616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-tmp\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:24.910701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.910667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhq6\" (UniqueName: \"kubernetes.io/projected/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-kube-api-access-5vhq6\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:24.929378 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.929352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:24.983531 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.983472    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rlsz7"]
Jul 28 14:30:24.986216 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.986200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:24.988625 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.988609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 28 14:30:24.988625 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.988618    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-72rzq\""
Jul 28 14:30:24.998958 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:24.998937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rlsz7"]
Jul 28 14:30:25.011183 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.011160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-tmp\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:25.011298 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.011208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhq6\" (UniqueName: \"kubernetes.io/projected/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-kube-api-access-5vhq6\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:25.011298 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.011244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-tls-certs\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:25.013836 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.013809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-tmp\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:25.013954 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.013932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-tls-certs\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:25.023588 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.023569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhq6\" (UniqueName: \"kubernetes.io/projected/a84f130a-8c89-4ebc-8088-a2ea767a4ff9-kube-api-access-5vhq6\") pod \"kube-auth-proxy-746f6cf6d8-2f9gn\" (UID: \"a84f130a-8c89-4ebc-8088-a2ea767a4ff9\") " pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:25.027533 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.027504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:25.109481 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.109451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:25.112431 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.112409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qfwg\" (UniqueName: \"kubernetes.io/projected/1c75c367-3583-405a-a265-f8dcdde207aa-kube-api-access-7qfwg\") pod \"odh-model-controller-7998c67dc4-rlsz7\" (UID: \"1c75c367-3583-405a-a265-f8dcdde207aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:25.112562 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.112440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c75c367-3583-405a-a265-f8dcdde207aa-cert\") pod \"odh-model-controller-7998c67dc4-rlsz7\" (UID: \"1c75c367-3583-405a-a265-f8dcdde207aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:25.154226 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.154189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"
Jul 28 14:30:25.210009 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.209978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:25.213636 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.213607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c75c367-3583-405a-a265-f8dcdde207aa-cert\") pod \"odh-model-controller-7998c67dc4-rlsz7\" (UID: \"1c75c367-3583-405a-a265-f8dcdde207aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:25.213761 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.213732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qfwg\" (UniqueName: \"kubernetes.io/projected/1c75c367-3583-405a-a265-f8dcdde207aa-kube-api-access-7qfwg\") pod \"odh-model-controller-7998c67dc4-rlsz7\" (UID: \"1c75c367-3583-405a-a265-f8dcdde207aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:25.216407 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.216365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c75c367-3583-405a-a265-f8dcdde207aa-cert\") pod \"odh-model-controller-7998c67dc4-rlsz7\" (UID: \"1c75c367-3583-405a-a265-f8dcdde207aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:25.222052 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.222000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qfwg\" (UniqueName: \"kubernetes.io/projected/1c75c367-3583-405a-a265-f8dcdde207aa-kube-api-access-7qfwg\") pod \"odh-model-controller-7998c67dc4-rlsz7\" (UID: \"1c75c367-3583-405a-a265-f8dcdde207aa\") " pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:25.282497 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.282469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn"]
Jul 28 14:30:25.284974 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:30:25.284941    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84f130a_8c89_4ebc_8088_a2ea767a4ff9.slice/crio-51c52f60275b9c296e99d1c3a049f2c2c7421f42fe3d9d8297d747d13917a8a4 WatchSource:0}: Error finding container 51c52f60275b9c296e99d1c3a049f2c2c7421f42fe3d9d8297d747d13917a8a4: Status 404 returned error can't find the container with id 51c52f60275b9c296e99d1c3a049f2c2c7421f42fe3d9d8297d747d13917a8a4
Jul 28 14:30:25.296314 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.296291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:25.434450 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.434422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rlsz7"]
Jul 28 14:30:25.439618 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:30:25.439593    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c75c367_3583_405a_a265_f8dcdde207aa.slice/crio-c83d60286481e5f5f8b80216af0ba12885ac8bd64510015f906ab737dab713d7 WatchSource:0}: Error finding container c83d60286481e5f5f8b80216af0ba12885ac8bd64510015f906ab737dab713d7: Status 404 returned error can't find the container with id c83d60286481e5f5f8b80216af0ba12885ac8bd64510015f906ab737dab713d7
Jul 28 14:30:25.619857 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.619828    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d4bl7"]
Jul 28 14:30:25.623735 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.623717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:25.626167 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.626147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 28 14:30:25.635250 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.635227    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d4bl7"]
Jul 28 14:30:25.718014 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.717972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5d8j\" (UniqueName: \"kubernetes.io/projected/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-kube-api-access-k5d8j\") pod \"model-serving-api-54d89bd5fc-d4bl7\" (UID: \"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:25.718142 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.718105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-tls-certs\") pod \"model-serving-api-54d89bd5fc-d4bl7\" (UID: \"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:25.818903 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.818870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5d8j\" (UniqueName: \"kubernetes.io/projected/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-kube-api-access-k5d8j\") pod \"model-serving-api-54d89bd5fc-d4bl7\" (UID: \"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:25.819052 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.818931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-tls-certs\") pod \"model-serving-api-54d89bd5fc-d4bl7\" (UID: \"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:25.819052 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:30:25.819017    2579 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 14:30:25.819130 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:30:25.819072    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-tls-certs podName:6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127 nodeName:}" failed. No retries permitted until 2026-07-28 14:30:26.319056179 +0000 UTC m=+469.765905466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-tls-certs") pod "model-serving-api-54d89bd5fc-d4bl7" (UID: "6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127") : secret "model-serving-api-tls" not found
Jul 28 14:30:25.830059 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.830032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5d8j\" (UniqueName: \"kubernetes.io/projected/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-kube-api-access-k5d8j\") pod \"model-serving-api-54d89bd5fc-d4bl7\" (UID: \"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:25.924737 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.924644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn" event={"ID":"a84f130a-8c89-4ebc-8088-a2ea767a4ff9","Type":"ContainerStarted","Data":"51c52f60275b9c296e99d1c3a049f2c2c7421f42fe3d9d8297d747d13917a8a4"}
Jul 28 14:30:25.925873 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:25.925847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7" event={"ID":"1c75c367-3583-405a-a265-f8dcdde207aa","Type":"ContainerStarted","Data":"c83d60286481e5f5f8b80216af0ba12885ac8bd64510015f906ab737dab713d7"}
Jul 28 14:30:26.073309 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.073275    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"]
Jul 28 14:30:26.076369 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.076354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.080028 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.080003    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 14:30:26.080239 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.080217    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 14:30:26.080470 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.080452    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 14:30:26.080470 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.080460    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 14:30:26.082158 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.082139    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g8qcm\""
Jul 28 14:30:26.098957 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.098933    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"]
Jul 28 14:30:26.222570 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.222512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfwv\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-kube-api-access-8kfwv\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.222677 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.222571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.222677 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.222635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79902e-84bd-4c05-9ecf-41caa5105470-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.222677 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.222672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.222837 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.222712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.222837 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.222755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.222837 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.222790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.301157 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.301111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:26.324156 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.324307 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-tls-certs\") pod \"model-serving-api-54d89bd5fc-d4bl7\" (UID: \"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:26.324307 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.324307 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.324880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfwv\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-kube-api-access-8kfwv\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.324880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.324880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79902e-84bd-4c05-9ecf-41caa5105470-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.324880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.324880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.324827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.327368 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.327337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79902e-84bd-4c05-9ecf-41caa5105470-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.327584 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.327564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.328277 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.328256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127-tls-certs\") pod \"model-serving-api-54d89bd5fc-d4bl7\" (UID: \"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127\") " pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:26.328718 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.328677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.329474 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.329452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.333796 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.333753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.334237 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.334201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfwv\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-kube-api-access-8kfwv\") pod \"istiod-openshift-gateway-7b7995d5cd-sx8kg\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.358758 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.358728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:26.385449 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.385427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:26.540880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.540844    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:26.563804 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.559217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"]
Jul 28 14:30:26.569140 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:30:26.569102    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb79902e_84bd_4c05_9ecf_41caa5105470.slice/crio-5417fc91c2a491f9e51ba7041511be444e7e09cbd87d7dbd60eab864bbd53bf7 WatchSource:0}: Error finding container 5417fc91c2a491f9e51ba7041511be444e7e09cbd87d7dbd60eab864bbd53bf7: Status 404 returned error can't find the container with id 5417fc91c2a491f9e51ba7041511be444e7e09cbd87d7dbd60eab864bbd53bf7
Jul 28 14:30:26.727379 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.727152    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-d4bl7"]
Jul 28 14:30:26.730199 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:30:26.730167    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0e5bfb_9881_4d8e_a8d2_4b1a7d9c9127.slice/crio-f85fd37b2f206ba3995394d717bdcda0275ecf6305ce061461c5af4050c7656d WatchSource:0}: Error finding container f85fd37b2f206ba3995394d717bdcda0275ecf6305ce061461c5af4050c7656d: Status 404 returned error can't find the container with id f85fd37b2f206ba3995394d717bdcda0275ecf6305ce061461c5af4050c7656d
Jul 28 14:30:26.933412 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.933354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7" event={"ID":"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127","Type":"ContainerStarted","Data":"f85fd37b2f206ba3995394d717bdcda0275ecf6305ce061461c5af4050c7656d"}
Jul 28 14:30:26.935018 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:26.934986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg" event={"ID":"cb79902e-84bd-4c05-9ecf-41caa5105470","Type":"ContainerStarted","Data":"5417fc91c2a491f9e51ba7041511be444e7e09cbd87d7dbd60eab864bbd53bf7"}
Jul 28 14:30:27.206884 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:27.206792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:31.177076 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.177047    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:30:31.177479 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.177117    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:30:31.960039 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.960001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg" event={"ID":"cb79902e-84bd-4c05-9ecf-41caa5105470","Type":"ContainerStarted","Data":"eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7"}
Jul 28 14:30:31.960220 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.960088    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:31.961561 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.961525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7" event={"ID":"1c75c367-3583-405a-a265-f8dcdde207aa","Type":"ContainerStarted","Data":"9344e36d88803f37daa7daf92228fe783bc980be50a9327126c1b21b3a226a8c"}
Jul 28 14:30:31.961742 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.961622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:31.963237 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.963210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn" event={"ID":"a84f130a-8c89-4ebc-8088-a2ea767a4ff9","Type":"ContainerStarted","Data":"8656f4f2d7964a12967278b02385b3b7e2a11606fad4d15ffeaec0156cf5712c"}
Jul 28 14:30:31.964683 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.964660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7" event={"ID":"6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127","Type":"ContainerStarted","Data":"4a1972124e6cce46c79da50df78e6cda5e2094d1d072f39d326da3aea7ccc684"}
Jul 28 14:30:31.964787 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.964743    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:31.985098 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:31.985058    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg" podStartSLOduration=1.379780461 podStartE2EDuration="5.985046781s" totalImagesPullingTime="4.60526632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:31.983839206 +0000 UTC m=+475.430688512" watchObservedRunningTime="2026-07-28 14:30:31.985046781 +0000 UTC m=+475.431896088"
Jul 28 14:30:32.001134 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:32.001094    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7" podStartSLOduration=2.2552512 podStartE2EDuration="8.001079909s" totalImagesPullingTime="5.745828709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:32.001048035 +0000 UTC m=+475.447897368" watchObservedRunningTime="2026-07-28 14:30:32.001079909 +0000 UTC m=+475.447929207"
Jul 28 14:30:32.016560 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:32.016518    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7" podStartSLOduration=2.576143763 podStartE2EDuration="7.016502909s" totalImagesPullingTime="4.440359146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:32.01621835 +0000 UTC m=+475.463067671" watchObservedRunningTime="2026-07-28 14:30:32.016502909 +0000 UTC m=+475.463352222"
Jul 28 14:30:32.032915 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:32.032876    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-746f6cf6d8-2f9gn" podStartSLOduration=2.150272963 podStartE2EDuration="8.032863758s" totalImagesPullingTime="5.882590795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:32.032270093 +0000 UTC m=+475.479119426" watchObservedRunningTime="2026-07-28 14:30:32.032863758 +0000 UTC m=+475.479713067"
Jul 28 14:30:32.970493 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:32.970461    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:33.243600 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:33.243520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:33.353593 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:33.353559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:33.570195 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:33.570166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:35.922489 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:35.922421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:35.990847 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:35.990817    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"]
Jul 28 14:30:35.994265 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:35.994241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:35.994704 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:35.994682    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"]
Jul 28 14:30:35.997441 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:35.997420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bl258\""
Jul 28 14:30:35.997557 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:35.997445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 28 14:30:35.997632 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:35.997580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:35.999912 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:35.999896    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 28 14:30:36.000148 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.000131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wdbgw\""
Jul 28 14:30:36.005762 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.005744    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"]
Jul 28 14:30:36.010405 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.010293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"]
Jul 28 14:30:36.099944 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.099916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:36.108997 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.108974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrg4\" (UniqueName: \"kubernetes.io/projected/b0574a69-64d6-472c-88cf-e616bb771f9f-kube-api-access-vkrg4\") pod \"llmisvc-controller-manager-777c6bd7d5-5zcc4\" (UID: \"b0574a69-64d6-472c-88cf-e616bb771f9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:36.109066 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.109009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpnln\" (UniqueName: \"kubernetes.io/projected/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-kube-api-access-kpnln\") pod \"kserve-controller-manager-d6f4b65b8-5nmq6\" (UID: \"8cbe6dc3-8ae8-408c-bfba-0a50737c5708\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:36.109066 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.109028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0574a69-64d6-472c-88cf-e616bb771f9f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5zcc4\" (UID: \"b0574a69-64d6-472c-88cf-e616bb771f9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:36.109137 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.109111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-cert\") pod \"kserve-controller-manager-d6f4b65b8-5nmq6\" (UID: \"8cbe6dc3-8ae8-408c-bfba-0a50737c5708\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:36.203671 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.203586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:36.209495 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.209456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrg4\" (UniqueName: \"kubernetes.io/projected/b0574a69-64d6-472c-88cf-e616bb771f9f-kube-api-access-vkrg4\") pod \"llmisvc-controller-manager-777c6bd7d5-5zcc4\" (UID: \"b0574a69-64d6-472c-88cf-e616bb771f9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:36.209663 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.209563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnln\" (UniqueName: \"kubernetes.io/projected/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-kube-api-access-kpnln\") pod \"kserve-controller-manager-d6f4b65b8-5nmq6\" (UID: \"8cbe6dc3-8ae8-408c-bfba-0a50737c5708\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:36.209663 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.209635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0574a69-64d6-472c-88cf-e616bb771f9f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5zcc4\" (UID: \"b0574a69-64d6-472c-88cf-e616bb771f9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:36.210091 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.209764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-cert\") pod \"kserve-controller-manager-d6f4b65b8-5nmq6\" (UID: \"8cbe6dc3-8ae8-408c-bfba-0a50737c5708\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:36.210091 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:30:36.209880    2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 14:30:36.210091 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:30:36.209950    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-cert podName:8cbe6dc3-8ae8-408c-bfba-0a50737c5708 nodeName:}" failed. No retries permitted until 2026-07-28 14:30:36.709932035 +0000 UTC m=+480.156781324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-cert") pod "kserve-controller-manager-d6f4b65b8-5nmq6" (UID: "8cbe6dc3-8ae8-408c-bfba-0a50737c5708") : secret "kserve-webhook-server-cert" not found
Jul 28 14:30:36.212696 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.212679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0574a69-64d6-472c-88cf-e616bb771f9f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5zcc4\" (UID: \"b0574a69-64d6-472c-88cf-e616bb771f9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:36.226342 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.226314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnln\" (UniqueName: \"kubernetes.io/projected/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-kube-api-access-kpnln\") pod \"kserve-controller-manager-d6f4b65b8-5nmq6\" (UID: \"8cbe6dc3-8ae8-408c-bfba-0a50737c5708\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:36.227158 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.227139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrg4\" (UniqueName: \"kubernetes.io/projected/b0574a69-64d6-472c-88cf-e616bb771f9f-kube-api-access-vkrg4\") pod \"llmisvc-controller-manager-777c6bd7d5-5zcc4\" (UID: \"b0574a69-64d6-472c-88cf-e616bb771f9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:36.314665 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.314647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:36.355232 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.355195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:36.456646 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.456625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"]
Jul 28 14:30:36.458581 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:30:36.458553    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb0574a69_64d6_472c_88cf_e616bb771f9f.slice/crio-7591ea1d98fbd0da6126e5b2a04e1dff901338e4e36c41b650f67fe61106a9cd WatchSource:0}: Error finding container 7591ea1d98fbd0da6126e5b2a04e1dff901338e4e36c41b650f67fe61106a9cd: Status 404 returned error can't find the container with id 7591ea1d98fbd0da6126e5b2a04e1dff901338e4e36c41b650f67fe61106a9cd
Jul 28 14:30:36.479058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.479034    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:30:36.714521 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.714463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-cert\") pod \"kserve-controller-manager-d6f4b65b8-5nmq6\" (UID: \"8cbe6dc3-8ae8-408c-bfba-0a50737c5708\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:36.716652 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.716633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cbe6dc3-8ae8-408c-bfba-0a50737c5708-cert\") pod \"kserve-controller-manager-d6f4b65b8-5nmq6\" (UID: \"8cbe6dc3-8ae8-408c-bfba-0a50737c5708\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:36.907428 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.907386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:36.987941 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:36.987881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4" event={"ID":"b0574a69-64d6-472c-88cf-e616bb771f9f","Type":"ContainerStarted","Data":"7591ea1d98fbd0da6126e5b2a04e1dff901338e4e36c41b650f67fe61106a9cd"}
Jul 28 14:30:37.037317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:37.037297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"]
Jul 28 14:30:37.039454 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:30:37.039429    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbe6dc3_8ae8_408c_bfba_0a50737c5708.slice/crio-14fbf5ef479dd9a54ae1e83ea261890640cc971b9963cdf288c5bb268934fb9c WatchSource:0}: Error finding container 14fbf5ef479dd9a54ae1e83ea261890640cc971b9963cdf288c5bb268934fb9c: Status 404 returned error can't find the container with id 14fbf5ef479dd9a54ae1e83ea261890640cc971b9963cdf288c5bb268934fb9c
Jul 28 14:30:37.996176 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:37.996067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6" event={"ID":"8cbe6dc3-8ae8-408c-bfba-0a50737c5708","Type":"ContainerStarted","Data":"14fbf5ef479dd9a54ae1e83ea261890640cc971b9963cdf288c5bb268934fb9c"}
Jul 28 14:30:40.005460 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:40.005426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6" event={"ID":"8cbe6dc3-8ae8-408c-bfba-0a50737c5708","Type":"ContainerStarted","Data":"90a8b421aaed9c43615e506e28bedb08893cb268befa285de586714c1cc2756e"}
Jul 28 14:30:40.005843 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:40.005569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:30:40.006913 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:40.006893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4" event={"ID":"b0574a69-64d6-472c-88cf-e616bb771f9f","Type":"ContainerStarted","Data":"1f92d2f96c45d3f31baaf8e461666b43ab89260e6d59170e55a40ff510e8307c"}
Jul 28 14:30:40.007036 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:40.007015    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:30:40.023335 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:40.023298    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6" podStartSLOduration=2.2278191019999998 podStartE2EDuration="5.023286363s" totalImagesPullingTime="2.795467261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:40.021205703 +0000 UTC m=+483.468055021" watchObservedRunningTime="2026-07-28 14:30:40.023286363 +0000 UTC m=+483.470135671"
Jul 28 14:30:40.054988 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:40.054947    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4" podStartSLOduration=1.6802951689999999 podStartE2EDuration="5.054936117s" totalImagesPullingTime="3.374640948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:30:40.053310048 +0000 UTC m=+483.500159356" watchObservedRunningTime="2026-07-28 14:30:40.054936117 +0000 UTC m=+483.501785427"
Jul 28 14:30:42.971877 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:42.971847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rlsz7"
Jul 28 14:30:42.973812 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:42.973795    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-d4bl7"
Jul 28 14:30:50.950979 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:50.950936    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"]
Jul 28 14:30:50.951482 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:50.951187    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg" podUID="cb79902e-84bd-4c05-9ecf-41caa5105470" containerName="discovery" containerID="cri-o://eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7" gracePeriod=30
Jul 28 14:30:50.969471 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:50.969443    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-sx8kg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.19:8080/ready\": dial tcp 10.132.0.19:8080: connect: connection refused" start-of-body=
Jul 28 14:30:50.969598 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:50.969510    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg" podUID="cb79902e-84bd-4c05-9ecf-41caa5105470" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.19:8080/ready\": dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 28 14:30:51.188334 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.188310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:51.319518 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.319492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-kubeconfig\") pod \"cb79902e-84bd-4c05-9ecf-41caa5105470\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") "
Jul 28 14:30:51.319681 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.319553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79902e-84bd-4c05-9ecf-41caa5105470-local-certs\") pod \"cb79902e-84bd-4c05-9ecf-41caa5105470\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") "
Jul 28 14:30:51.319681 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.319589    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kfwv\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-kube-api-access-8kfwv\") pod \"cb79902e-84bd-4c05-9ecf-41caa5105470\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") "
Jul 28 14:30:51.319790 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.319678    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-dns-cert\") pod \"cb79902e-84bd-4c05-9ecf-41caa5105470\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") "
Jul 28 14:30:51.319790 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.319706    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-ca-configmap\") pod \"cb79902e-84bd-4c05-9ecf-41caa5105470\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") "
Jul 28 14:30:51.319790 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.319754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-token\") pod \"cb79902e-84bd-4c05-9ecf-41caa5105470\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") "
Jul 28 14:30:51.319790 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.319784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-cacerts\") pod \"cb79902e-84bd-4c05-9ecf-41caa5105470\" (UID: \"cb79902e-84bd-4c05-9ecf-41caa5105470\") "
Jul 28 14:30:51.320106 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.320065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cb79902e-84bd-4c05-9ecf-41caa5105470" (UID: "cb79902e-84bd-4c05-9ecf-41caa5105470"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:30:51.321858 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.321822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb79902e-84bd-4c05-9ecf-41caa5105470-local-certs" (OuterVolumeSpecName: "local-certs") pod "cb79902e-84bd-4c05-9ecf-41caa5105470" (UID: "cb79902e-84bd-4c05-9ecf-41caa5105470"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:30:51.321984 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.321961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-cacerts" (OuterVolumeSpecName: "cacerts") pod "cb79902e-84bd-4c05-9ecf-41caa5105470" (UID: "cb79902e-84bd-4c05-9ecf-41caa5105470"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:30:51.322129 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.322097    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-kube-api-access-8kfwv" (OuterVolumeSpecName: "kube-api-access-8kfwv") pod "cb79902e-84bd-4c05-9ecf-41caa5105470" (UID: "cb79902e-84bd-4c05-9ecf-41caa5105470"). InnerVolumeSpecName "kube-api-access-8kfwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:30:51.322215 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.322127    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb79902e-84bd-4c05-9ecf-41caa5105470" (UID: "cb79902e-84bd-4c05-9ecf-41caa5105470"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:30:51.322215 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.322147    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cb79902e-84bd-4c05-9ecf-41caa5105470" (UID: "cb79902e-84bd-4c05-9ecf-41caa5105470"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:30:51.322215 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.322184    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cb79902e-84bd-4c05-9ecf-41caa5105470" (UID: "cb79902e-84bd-4c05-9ecf-41caa5105470"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:30:51.421204 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.421179    2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb79902e-84bd-4c05-9ecf-41caa5105470-local-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:30:51.421204 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.421205    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kfwv\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-kube-api-access-8kfwv\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:30:51.421327 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.421218    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-dns-cert\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:30:51.421327 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.421231    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-csr-ca-configmap\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:30:51.421327 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.421243    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-token\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:30:51.421327 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.421255    2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-cacerts\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:30:51.421327 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:51.421266    2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cb79902e-84bd-4c05-9ecf-41caa5105470-istio-kubeconfig\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:30:52.055773 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.055740    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb79902e-84bd-4c05-9ecf-41caa5105470" containerID="eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7" exitCode=0
Jul 28 14:30:52.056232 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.055803    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"
Jul 28 14:30:52.056232 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.055830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg" event={"ID":"cb79902e-84bd-4c05-9ecf-41caa5105470","Type":"ContainerDied","Data":"eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7"}
Jul 28 14:30:52.056232 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.055903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg" event={"ID":"cb79902e-84bd-4c05-9ecf-41caa5105470","Type":"ContainerDied","Data":"5417fc91c2a491f9e51ba7041511be444e7e09cbd87d7dbd60eab864bbd53bf7"}
Jul 28 14:30:52.056232 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.055925    2579 scope.go:117] "RemoveContainer" containerID="eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7"
Jul 28 14:30:52.064882 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.064861    2579 scope.go:117] "RemoveContainer" containerID="eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7"
Jul 28 14:30:52.065153 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:30:52.065130    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7\": container with ID starting with eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7 not found: ID does not exist" containerID="eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7"
Jul 28 14:30:52.065197 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.065161    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7"} err="failed to get container status \"eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7\": rpc error: code = NotFound desc = could not find container \"eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7\": container with ID starting with eeccb68f741a2c8fd45118ceec6ac81a95f0f8950cfad035c069a84b5e8d49d7 not found: ID does not exist"
Jul 28 14:30:52.076617 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.076591    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"]
Jul 28 14:30:52.083029 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:52.083009    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-sx8kg"]
Jul 28 14:30:53.251839 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:30:53.251804    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb79902e-84bd-4c05-9ecf-41caa5105470" path="/var/lib/kubelet/pods/cb79902e-84bd-4c05-9ecf-41caa5105470/volumes"
Jul 28 14:31:11.012239 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:11.012207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5zcc4"
Jul 28 14:31:11.015166 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:11.015147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5nmq6"
Jul 28 14:31:25.848841 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:25.848789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:25.953101 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:25.953065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:25.991412 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:25.991363    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cr72"]
Jul 28 14:31:25.991667 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:25.991648    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb79902e-84bd-4c05-9ecf-41caa5105470" containerName="discovery"
Jul 28 14:31:25.991712 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:25.991669    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb79902e-84bd-4c05-9ecf-41caa5105470" containerName="discovery"
Jul 28 14:31:25.991746 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:25.991730    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb79902e-84bd-4c05-9ecf-41caa5105470" containerName="discovery"
Jul 28 14:31:25.996096 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:25.996077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2cr72"
Jul 28 14:31:26.002043 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.002019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-fjb74\""
Jul 28 14:31:26.002939 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.002913    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 14:31:26.003279 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.003259    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 14:31:26.010880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.010854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cr72"]
Jul 28 14:31:26.053654 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.053626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7w6\" (UniqueName: \"kubernetes.io/projected/c6a011cd-ced1-495f-ba57-7a79ab6f1914-kube-api-access-qr7w6\") pod \"kuadrant-operator-catalog-2cr72\" (UID: \"c6a011cd-ced1-495f-ba57-7a79ab6f1914\") " pod="kuadrant-system/kuadrant-operator-catalog-2cr72"
Jul 28 14:31:26.154749 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.154658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7w6\" (UniqueName: \"kubernetes.io/projected/c6a011cd-ced1-495f-ba57-7a79ab6f1914-kube-api-access-qr7w6\") pod \"kuadrant-operator-catalog-2cr72\" (UID: \"c6a011cd-ced1-495f-ba57-7a79ab6f1914\") " pod="kuadrant-system/kuadrant-operator-catalog-2cr72"
Jul 28 14:31:26.162934 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.162897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7w6\" (UniqueName: \"kubernetes.io/projected/c6a011cd-ced1-495f-ba57-7a79ab6f1914-kube-api-access-qr7w6\") pod \"kuadrant-operator-catalog-2cr72\" (UID: \"c6a011cd-ced1-495f-ba57-7a79ab6f1914\") " pod="kuadrant-system/kuadrant-operator-catalog-2cr72"
Jul 28 14:31:26.177906 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.177864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:26.306850 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.306811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2cr72"
Jul 28 14:31:26.354135 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.353992    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cr72"]
Jul 28 14:31:26.434647 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.434573    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cr72"]
Jul 28 14:31:26.439007 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:31:26.438982    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a011cd_ced1_495f_ba57_7a79ab6f1914.slice/crio-f891c2ecf6fd06da3beb3ad4b7e229a964c541eec07bdb8c2fac4332dd5af612 WatchSource:0}: Error finding container f891c2ecf6fd06da3beb3ad4b7e229a964c541eec07bdb8c2fac4332dd5af612: Status 404 returned error can't find the container with id f891c2ecf6fd06da3beb3ad4b7e229a964c541eec07bdb8c2fac4332dd5af612
Jul 28 14:31:26.564819 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.564781    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fxk7p"]
Jul 28 14:31:26.567575 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.567553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:26.575794 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.575769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fxk7p"]
Jul 28 14:31:26.658054 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.658024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrz2h\" (UniqueName: \"kubernetes.io/projected/c29431f9-e60b-4c91-b9ef-77000f3ce73c-kube-api-access-jrz2h\") pod \"kuadrant-operator-catalog-fxk7p\" (UID: \"c29431f9-e60b-4c91-b9ef-77000f3ce73c\") " pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:26.759015 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.758928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrz2h\" (UniqueName: \"kubernetes.io/projected/c29431f9-e60b-4c91-b9ef-77000f3ce73c-kube-api-access-jrz2h\") pod \"kuadrant-operator-catalog-fxk7p\" (UID: \"c29431f9-e60b-4c91-b9ef-77000f3ce73c\") " pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:26.770689 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.770652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrz2h\" (UniqueName: \"kubernetes.io/projected/c29431f9-e60b-4c91-b9ef-77000f3ce73c-kube-api-access-jrz2h\") pod \"kuadrant-operator-catalog-fxk7p\" (UID: \"c29431f9-e60b-4c91-b9ef-77000f3ce73c\") " pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:26.834473 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.834437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:26.877415 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:26.877355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:27.011520 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:27.011494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fxk7p"]
Jul 28 14:31:27.020943 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:31:27.020912    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29431f9_e60b_4c91_b9ef_77000f3ce73c.slice/crio-a1d1d3a95081e5e3f97c5ce64ce8bab963d902ba9eb4451a10ae63dde22ca00d WatchSource:0}: Error finding container a1d1d3a95081e5e3f97c5ce64ce8bab963d902ba9eb4451a10ae63dde22ca00d: Status 404 returned error can't find the container with id a1d1d3a95081e5e3f97c5ce64ce8bab963d902ba9eb4451a10ae63dde22ca00d
Jul 28 14:31:27.196012 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:27.195973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p" event={"ID":"c29431f9-e60b-4c91-b9ef-77000f3ce73c","Type":"ContainerStarted","Data":"a1d1d3a95081e5e3f97c5ce64ce8bab963d902ba9eb4451a10ae63dde22ca00d"}
Jul 28 14:31:27.197265 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:27.197239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2cr72" event={"ID":"c6a011cd-ced1-495f-ba57-7a79ab6f1914","Type":"ContainerStarted","Data":"f891c2ecf6fd06da3beb3ad4b7e229a964c541eec07bdb8c2fac4332dd5af612"}
Jul 28 14:31:27.606279 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:27.606243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:29.207659 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.207564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2cr72" event={"ID":"c6a011cd-ced1-495f-ba57-7a79ab6f1914","Type":"ContainerStarted","Data":"e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d"}
Jul 28 14:31:29.208102 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.207629    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2cr72" podUID="c6a011cd-ced1-495f-ba57-7a79ab6f1914" containerName="registry-server" containerID="cri-o://e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d" gracePeriod=2
Jul 28 14:31:29.209009 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.208987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p" event={"ID":"c29431f9-e60b-4c91-b9ef-77000f3ce73c","Type":"ContainerStarted","Data":"3919a113e9131b21b2ede39b728c99ce89f365702472203c865c23f7ea185617"}
Jul 28 14:31:29.224485 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.224439    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2cr72" podStartSLOduration=2.312868164 podStartE2EDuration="4.224423851s" totalImagesPullingTime="1.911555687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:31:29.222542734 +0000 UTC m=+532.669392044" watchObservedRunningTime="2026-07-28 14:31:29.224423851 +0000 UTC m=+532.671273160"
Jul 28 14:31:29.238518 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.238475    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p" podStartSLOduration=1.9078531239999998 podStartE2EDuration="3.238461972s" totalImagesPullingTime="1.330608848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:31:29.236880659 +0000 UTC m=+532.683729968" watchObservedRunningTime="2026-07-28 14:31:29.238461972 +0000 UTC m=+532.685311278"
Jul 28 14:31:29.365541 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.365509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:29.447478 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.447452    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2cr72"
Jul 28 14:31:29.481085 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.481018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr7w6\" (UniqueName: \"kubernetes.io/projected/c6a011cd-ced1-495f-ba57-7a79ab6f1914-kube-api-access-qr7w6\") pod \"c6a011cd-ced1-495f-ba57-7a79ab6f1914\" (UID: \"c6a011cd-ced1-495f-ba57-7a79ab6f1914\") "
Jul 28 14:31:29.483099 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.483066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a011cd-ced1-495f-ba57-7a79ab6f1914-kube-api-access-qr7w6" (OuterVolumeSpecName: "kube-api-access-qr7w6") pod "c6a011cd-ced1-495f-ba57-7a79ab6f1914" (UID: "c6a011cd-ced1-495f-ba57-7a79ab6f1914"). InnerVolumeSpecName "kube-api-access-qr7w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:31:29.518987 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.518959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:29.574100 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.574065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:29.581885 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:29.581860    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr7w6\" (UniqueName: \"kubernetes.io/projected/c6a011cd-ced1-495f-ba57-7a79ab6f1914-kube-api-access-qr7w6\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:31:30.122971 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.122934    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:30.180410 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.180371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:30.214134 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.214101    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6a011cd-ced1-495f-ba57-7a79ab6f1914" containerID="e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d" exitCode=0
Jul 28 14:31:30.214563 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.214143    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2cr72"
Jul 28 14:31:30.214563 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.214194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2cr72" event={"ID":"c6a011cd-ced1-495f-ba57-7a79ab6f1914","Type":"ContainerDied","Data":"e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d"}
Jul 28 14:31:30.214563 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.214245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2cr72" event={"ID":"c6a011cd-ced1-495f-ba57-7a79ab6f1914","Type":"ContainerDied","Data":"f891c2ecf6fd06da3beb3ad4b7e229a964c541eec07bdb8c2fac4332dd5af612"}
Jul 28 14:31:30.214563 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.214274    2579 scope.go:117] "RemoveContainer" containerID="e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d"
Jul 28 14:31:30.223075 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.223058    2579 scope.go:117] "RemoveContainer" containerID="e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d"
Jul 28 14:31:30.223324 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:31:30.223305    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d\": container with ID starting with e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d not found: ID does not exist" containerID="e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d"
Jul 28 14:31:30.223387 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.223332    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d"} err="failed to get container status \"e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d\": rpc error: code = NotFound desc = could not find container \"e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d\": container with ID starting with e312464365ae5035659beb9d26c57cc9ccea2aa37933095a47c260e682cd434d not found: ID does not exist"
Jul 28 14:31:30.234249 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.234229    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cr72"]
Jul 28 14:31:30.238286 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:30.238265    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2cr72"]
Jul 28 14:31:31.250916 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:31.250875    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a011cd-ced1-495f-ba57-7a79ab6f1914" path="/var/lib/kubelet/pods/c6a011cd-ced1-495f-ba57-7a79ab6f1914/volumes"
Jul 28 14:31:36.878215 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:36.878175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:36.878646 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:36.878241    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:36.899621 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:36.899600    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:37.265407 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:37.265329    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-fxk7p"
Jul 28 14:31:40.536358 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:40.536313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:40.637282 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:40.637243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:54.721935 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:54.721892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:31:57.356930 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.356895    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"]
Jul 28 14:31:57.357314 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.357164    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a011cd-ced1-495f-ba57-7a79ab6f1914" containerName="registry-server"
Jul 28 14:31:57.357314 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.357175    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a011cd-ced1-495f-ba57-7a79ab6f1914" containerName="registry-server"
Jul 28 14:31:57.357314 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.357227    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a011cd-ced1-495f-ba57-7a79ab6f1914" containerName="registry-server"
Jul 28 14:31:57.363542 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.363520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"
Jul 28 14:31:57.366325 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.366306    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-d2mhg\""
Jul 28 14:31:57.367804 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.367786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 14:31:57.373811 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.373787    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"]
Jul 28 14:31:57.487858 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.487822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2gj9\" (UniqueName: \"kubernetes.io/projected/42a9eea7-1c10-4b86-8f1a-f06ed03800b3-kube-api-access-d2gj9\") pod \"dns-operator-controller-manager-864dd78c4c-svjl9\" (UID: \"42a9eea7-1c10-4b86-8f1a-f06ed03800b3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"
Jul 28 14:31:57.588797 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.588751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2gj9\" (UniqueName: \"kubernetes.io/projected/42a9eea7-1c10-4b86-8f1a-f06ed03800b3-kube-api-access-d2gj9\") pod \"dns-operator-controller-manager-864dd78c4c-svjl9\" (UID: \"42a9eea7-1c10-4b86-8f1a-f06ed03800b3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"
Jul 28 14:31:57.596927 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.596900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2gj9\" (UniqueName: \"kubernetes.io/projected/42a9eea7-1c10-4b86-8f1a-f06ed03800b3-kube-api-access-d2gj9\") pod \"dns-operator-controller-manager-864dd78c4c-svjl9\" (UID: \"42a9eea7-1c10-4b86-8f1a-f06ed03800b3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"
Jul 28 14:31:57.673884 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.673808    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"
Jul 28 14:31:57.819964 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:57.819941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"]
Jul 28 14:31:57.822589 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:31:57.822564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a9eea7_1c10_4b86_8f1a_f06ed03800b3.slice/crio-21a347314bac73a2b97ba8fef6dbc904308dd7798b273baadc5ec6c424bcfd7a WatchSource:0}: Error finding container 21a347314bac73a2b97ba8fef6dbc904308dd7798b273baadc5ec6c424bcfd7a: Status 404 returned error can't find the container with id 21a347314bac73a2b97ba8fef6dbc904308dd7798b273baadc5ec6c424bcfd7a
Jul 28 14:31:58.321183 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:58.321147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9" event={"ID":"42a9eea7-1c10-4b86-8f1a-f06ed03800b3","Type":"ContainerStarted","Data":"21a347314bac73a2b97ba8fef6dbc904308dd7798b273baadc5ec6c424bcfd7a"}
Jul 28 14:31:58.924057 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:31:58.923940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:00.751770 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:00.751687    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"]
Jul 28 14:32:00.756423 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:00.756289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:00.759498 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:00.759476    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-c2xfs\""
Jul 28 14:32:00.766173 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:00.766147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"]
Jul 28 14:32:00.813748 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:00.813707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvvx\" (UniqueName: \"kubernetes.io/projected/6cdb6b34-c212-4425-90db-e8aee678b1d5-kube-api-access-ctvvx\") pod \"limitador-operator-controller-manager-859d7b78bd-sk4t9\" (UID: \"6cdb6b34-c212-4425-90db-e8aee678b1d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:00.914738 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:00.914690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvvx\" (UniqueName: \"kubernetes.io/projected/6cdb6b34-c212-4425-90db-e8aee678b1d5-kube-api-access-ctvvx\") pod \"limitador-operator-controller-manager-859d7b78bd-sk4t9\" (UID: \"6cdb6b34-c212-4425-90db-e8aee678b1d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:00.935179 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:00.935155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvvx\" (UniqueName: \"kubernetes.io/projected/6cdb6b34-c212-4425-90db-e8aee678b1d5-kube-api-access-ctvvx\") pod \"limitador-operator-controller-manager-859d7b78bd-sk4t9\" (UID: \"6cdb6b34-c212-4425-90db-e8aee678b1d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:01.067402 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:01.067348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:01.200594 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:01.200562    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"]
Jul 28 14:32:01.204569 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:01.204533    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdb6b34_c212_4425_90db_e8aee678b1d5.slice/crio-d1f33d642c03e96ef253232abbb17764277c5dc4cf4b52f2b016b174ea2857ca WatchSource:0}: Error finding container d1f33d642c03e96ef253232abbb17764277c5dc4cf4b52f2b016b174ea2857ca: Status 404 returned error can't find the container with id d1f33d642c03e96ef253232abbb17764277c5dc4cf4b52f2b016b174ea2857ca
Jul 28 14:32:01.333775 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:01.333688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9" event={"ID":"6cdb6b34-c212-4425-90db-e8aee678b1d5","Type":"ContainerStarted","Data":"d1f33d642c03e96ef253232abbb17764277c5dc4cf4b52f2b016b174ea2857ca"}
Jul 28 14:32:01.335274 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:01.335250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9" event={"ID":"42a9eea7-1c10-4b86-8f1a-f06ed03800b3","Type":"ContainerStarted","Data":"8d83ab89d491ce12aa29b13fc3507d7292ee2f3891e9f709a223d34a87bfb8c3"}
Jul 28 14:32:01.335419 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:01.335404    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"
Jul 28 14:32:01.364830 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:01.364784    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9" podStartSLOduration=1.86886402 podStartE2EDuration="4.364769835s" totalImagesPullingTime="2.495905815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:01.363158449 +0000 UTC m=+564.810007805" watchObservedRunningTime="2026-07-28 14:32:01.364769835 +0000 UTC m=+564.811619144"
Jul 28 14:32:01.366607 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:01.366586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:01.522558 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:01.522522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:02.319831 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:02.319788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:02.799878 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:02.799842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:03.346724 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:03.346629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9" event={"ID":"6cdb6b34-c212-4425-90db-e8aee678b1d5","Type":"ContainerStarted","Data":"cbb9e8386439c4ef97a79b676732ef54d90ae2ac1ca2ef419ba9a4cd7f0d4962"}
Jul 28 14:32:03.346724 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:03.346711    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:03.374747 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:03.374700    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9" podStartSLOduration=1.555721829 podStartE2EDuration="3.37468654s" totalImagesPullingTime="1.818964711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:03.373413347 +0000 UTC m=+566.820262657" watchObservedRunningTime="2026-07-28 14:32:03.37468654 +0000 UTC m=+566.821535849"
Jul 28 14:32:05.564371 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.564334    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"]
Jul 28 14:32:05.566469 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.566451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:05.569385 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.569363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sr8xr\""
Jul 28 14:32:05.585805 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.585780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"]
Jul 28 14:32:05.654711 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.654675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" (UID: \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:05.654883 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.654799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vlz\" (UniqueName: \"kubernetes.io/projected/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-kube-api-access-55vlz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" (UID: \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:05.755186 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.755139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vlz\" (UniqueName: \"kubernetes.io/projected/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-kube-api-access-55vlz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" (UID: \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:05.755348 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.755198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" (UID: \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:05.755659 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.755637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" (UID: \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:05.768218 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.768189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vlz\" (UniqueName: \"kubernetes.io/projected/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-kube-api-access-55vlz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" (UID: \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:05.875786 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.875706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:05.932103 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:05.932070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:06.005463 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:06.005435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"]
Jul 28 14:32:06.009320 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:06.009288    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc366c25c_6859_4bf5_b2f8_7fa7a3077b7b.slice/crio-e819fdb82e258f53da90921efcdebdd0ae954797d2014676751c739eb355b9e7 WatchSource:0}: Error finding container e819fdb82e258f53da90921efcdebdd0ae954797d2014676751c739eb355b9e7: Status 404 returned error can't find the container with id e819fdb82e258f53da90921efcdebdd0ae954797d2014676751c739eb355b9e7
Jul 28 14:32:06.320037 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:06.320002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:06.361002 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:06.360971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl" event={"ID":"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b","Type":"ContainerStarted","Data":"e819fdb82e258f53da90921efcdebdd0ae954797d2014676751c739eb355b9e7"}
Jul 28 14:32:06.587488 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:06.587414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:08.983467 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:08.983431    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"]
Jul 28 14:32:08.985619 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:08.985598    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"
Jul 28 14:32:08.989837 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:08.989819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9hk29\""
Jul 28 14:32:08.997534 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:08.997516    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"]
Jul 28 14:32:09.079904 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:09.079868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pv4j\" (UniqueName: \"kubernetes.io/projected/606d89cc-4b15-4a9d-8d0d-8c3af9006b2e-kube-api-access-8pv4j\") pod \"authorino-operator-75cc8d5cb7-vwjzn\" (UID: \"606d89cc-4b15-4a9d-8d0d-8c3af9006b2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"
Jul 28 14:32:09.180751 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:09.180715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pv4j\" (UniqueName: \"kubernetes.io/projected/606d89cc-4b15-4a9d-8d0d-8c3af9006b2e-kube-api-access-8pv4j\") pod \"authorino-operator-75cc8d5cb7-vwjzn\" (UID: \"606d89cc-4b15-4a9d-8d0d-8c3af9006b2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"
Jul 28 14:32:09.190673 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:09.190645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pv4j\" (UniqueName: \"kubernetes.io/projected/606d89cc-4b15-4a9d-8d0d-8c3af9006b2e-kube-api-access-8pv4j\") pod \"authorino-operator-75cc8d5cb7-vwjzn\" (UID: \"606d89cc-4b15-4a9d-8d0d-8c3af9006b2e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"
Jul 28 14:32:09.295605 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:09.295584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"
Jul 28 14:32:09.363346 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:09.363263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:09.442071 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:09.442042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"]
Jul 28 14:32:09.444714 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:09.444685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606d89cc_4b15_4a9d_8d0d_8c3af9006b2e.slice/crio-4cdd8d1e4c9f2a445b2eb31c8a5001b28e6e9ffef859e01985dc05c21f715208 WatchSource:0}: Error finding container 4cdd8d1e4c9f2a445b2eb31c8a5001b28e6e9ffef859e01985dc05c21f715208: Status 404 returned error can't find the container with id 4cdd8d1e4c9f2a445b2eb31c8a5001b28e6e9ffef859e01985dc05c21f715208
Jul 28 14:32:10.382636 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:10.382594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn" event={"ID":"606d89cc-4b15-4a9d-8d0d-8c3af9006b2e","Type":"ContainerStarted","Data":"4cdd8d1e4c9f2a445b2eb31c8a5001b28e6e9ffef859e01985dc05c21f715208"}
Jul 28 14:32:10.729454 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:10.729344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:10.828351 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:10.828262    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:12.343120 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:12.343090    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-svjl9"
Jul 28 14:32:13.397772 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:13.397736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl" event={"ID":"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b","Type":"ContainerStarted","Data":"357a92af67c1d9a99a4739f65c2bd0c67ad6831aa086c49fc3b5d7cadf204724"}
Jul 28 14:32:13.398270 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:13.397840    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:13.399336 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:13.399311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn" event={"ID":"606d89cc-4b15-4a9d-8d0d-8c3af9006b2e","Type":"ContainerStarted","Data":"0a75c5d2826d25bc53309d831bb32d4990477c8791f8a03c34f69a713fc66931"}
Jul 28 14:32:13.399475 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:13.399378    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"
Jul 28 14:32:13.418516 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:13.418477    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl" podStartSLOduration=1.86710022 podStartE2EDuration="8.418465558s" totalImagesPullingTime="6.551365338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:13.417573051 +0000 UTC m=+576.864422359" watchObservedRunningTime="2026-07-28 14:32:13.418465558 +0000 UTC m=+576.865314867"
Jul 28 14:32:13.433613 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:13.433568    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn" podStartSLOduration=2.311264541 podStartE2EDuration="5.433554953s" totalImagesPullingTime="3.122290412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:13.433513773 +0000 UTC m=+576.880363094" watchObservedRunningTime="2026-07-28 14:32:13.433554953 +0000 UTC m=+576.880404255"
Jul 28 14:32:14.352783 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:14.352750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:14.530348 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:14.530297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:14.626231 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:14.626142    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:15.504001 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:15.503964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:15.765608 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:15.765523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:17.923006 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:17.922967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:17.980230 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:17.980197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:24.405222 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:24.405192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:24.405655 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:24.405258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vwjzn"
Jul 28 14:32:25.447749 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.447709    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"]
Jul 28 14:32:25.449786 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.449770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:25.464812 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.464786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"]
Jul 28 14:32:25.508684 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.508641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgr22\" (UniqueName: \"kubernetes.io/projected/731882f6-6766-40eb-b24d-c91fbffbf668-kube-api-access-xgr22\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" (UID: \"731882f6-6766-40eb-b24d-c91fbffbf668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:25.508684 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.508692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/731882f6-6766-40eb-b24d-c91fbffbf668-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" (UID: \"731882f6-6766-40eb-b24d-c91fbffbf668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:25.609070 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.609028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgr22\" (UniqueName: \"kubernetes.io/projected/731882f6-6766-40eb-b24d-c91fbffbf668-kube-api-access-xgr22\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" (UID: \"731882f6-6766-40eb-b24d-c91fbffbf668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:25.609243 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.609087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/731882f6-6766-40eb-b24d-c91fbffbf668-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" (UID: \"731882f6-6766-40eb-b24d-c91fbffbf668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:25.609600 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.609581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/731882f6-6766-40eb-b24d-c91fbffbf668-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" (UID: \"731882f6-6766-40eb-b24d-c91fbffbf668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:25.618278 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.618246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgr22\" (UniqueName: \"kubernetes.io/projected/731882f6-6766-40eb-b24d-c91fbffbf668-kube-api-access-xgr22\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" (UID: \"731882f6-6766-40eb-b24d-c91fbffbf668\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:25.761011 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.760929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:25.903978 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:25.903946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"]
Jul 28 14:32:25.906594 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:25.906568    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731882f6_6766_40eb_b24d_c91fbffbf668.slice/crio-5ea0f1935c3711c3dd71e895c94a628b3fae1635ff372fdb20ea57b7dc3afbf4 WatchSource:0}: Error finding container 5ea0f1935c3711c3dd71e895c94a628b3fae1635ff372fdb20ea57b7dc3afbf4: Status 404 returned error can't find the container with id 5ea0f1935c3711c3dd71e895c94a628b3fae1635ff372fdb20ea57b7dc3afbf4
Jul 28 14:32:26.210409 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.207241    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"]
Jul 28 14:32:26.219438 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.219378    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"]
Jul 28 14:32:26.235740 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.235706    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"]
Jul 28 14:32:26.235927 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.235899    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" containerName="manager" containerID="cri-o://357a92af67c1d9a99a4739f65c2bd0c67ad6831aa086c49fc3b5d7cadf204724" gracePeriod=2
Jul 28 14:32:26.252380 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.252333    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"]
Jul 28 14:32:26.254202 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.254165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"]
Jul 28 14:32:26.254561 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.254545    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" containerName="manager"
Jul 28 14:32:26.254614 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.254563    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" containerName="manager"
Jul 28 14:32:26.254614 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.254574    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" containerName="manager"
Jul 28 14:32:26.254614 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.254580    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" containerName="manager"
Jul 28 14:32:26.254701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.254629    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" containerName="manager"
Jul 28 14:32:26.254701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.254637    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" containerName="manager"
Jul 28 14:32:26.254989 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.254959    2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\": pods \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:26.256543 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.256522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:26.274883 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.274860    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"]
Jul 28 14:32:26.275049 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.275029    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" containerName="manager" containerID="cri-o://cbb9e8386439c4ef97a79b676732ef54d90ae2ac1ca2ef419ba9a4cd7f0d4962" gracePeriod=2
Jul 28 14:32:26.286093 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.286064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"]
Jul 28 14:32:26.287494 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.287472    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"]
Jul 28 14:32:26.289303 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.289283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:26.296443 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.296102    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"]
Jul 28 14:32:26.313811 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.313775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97a5535b-a9f6-462a-b15a-22c3a224a7cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l996t\" (UID: \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:26.313966 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.313879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpcz\" (UniqueName: \"kubernetes.io/projected/402e90a4-c7d5-4db1-ad88-44315e785a10-kube-api-access-rcpcz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6ck2r\" (UID: \"402e90a4-c7d5-4db1-ad88-44315e785a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:26.313966 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.313904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpsv\" (UniqueName: \"kubernetes.io/projected/97a5535b-a9f6-462a-b15a-22c3a224a7cd-kube-api-access-5cpsv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l996t\" (UID: \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:26.313966 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.313961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/402e90a4-c7d5-4db1-ad88-44315e785a10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6ck2r\" (UID: \"402e90a4-c7d5-4db1-ad88-44315e785a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:26.319789 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.319760    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"]
Jul 28 14:32:26.320220 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.320199    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" containerName="manager"
Jul 28 14:32:26.320220 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.320219    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" containerName="manager"
Jul 28 14:32:26.320447 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.320304    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" containerName="manager"
Jul 28 14:32:26.322107 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.322079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"
Jul 28 14:32:26.331158 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.331114    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:26.332178 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.332095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"]
Jul 28 14:32:26.333499 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.333438    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:26.334462 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.334441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"]
Jul 28 14:32:26.361233 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.361202    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sk4t9\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:26.415582 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.414781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/402e90a4-c7d5-4db1-ad88-44315e785a10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6ck2r\" (UID: \"402e90a4-c7d5-4db1-ad88-44315e785a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:26.415582 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.414931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97a5535b-a9f6-462a-b15a-22c3a224a7cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l996t\" (UID: \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:26.415582 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.415041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpcz\" (UniqueName: \"kubernetes.io/projected/402e90a4-c7d5-4db1-ad88-44315e785a10-kube-api-access-rcpcz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6ck2r\" (UID: \"402e90a4-c7d5-4db1-ad88-44315e785a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:26.415582 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.415069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpsv\" (UniqueName: \"kubernetes.io/projected/97a5535b-a9f6-462a-b15a-22c3a224a7cd-kube-api-access-5cpsv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l996t\" (UID: \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:26.415582 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.415299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/402e90a4-c7d5-4db1-ad88-44315e785a10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6ck2r\" (UID: \"402e90a4-c7d5-4db1-ad88-44315e785a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:26.415910 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.415827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97a5535b-a9f6-462a-b15a-22c3a224a7cd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l996t\" (UID: \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:26.423567 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.423228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:26.425260 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.425214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpcz\" (UniqueName: \"kubernetes.io/projected/402e90a4-c7d5-4db1-ad88-44315e785a10-kube-api-access-rcpcz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6ck2r\" (UID: \"402e90a4-c7d5-4db1-ad88-44315e785a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:26.425687 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.425669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpsv\" (UniqueName: \"kubernetes.io/projected/97a5535b-a9f6-462a-b15a-22c3a224a7cd-kube-api-access-5cpsv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l996t\" (UID: \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:26.463065 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.461761    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p" container="manager"
Jul 28 14:32:26.464288 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.464246    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sk4t9\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:26.464597 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.464571    2579 generic.go:356] "Generic (PLEG): container finished" podID="6cdb6b34-c212-4425-90db-e8aee678b1d5" containerID="cbb9e8386439c4ef97a79b676732ef54d90ae2ac1ca2ef419ba9a4cd7f0d4962" exitCode=0
Jul 28 14:32:26.466763 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.466728    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:26.467098 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.467080    2579 generic.go:356] "Generic (PLEG): container finished" podID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" containerID="357a92af67c1d9a99a4739f65c2bd0c67ad6831aa086c49fc3b5d7cadf204724" exitCode=0
Jul 28 14:32:26.468897 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.468872    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:26.508127 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.508097    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:26.510331 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.510296    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:26.512027 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.511997    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:26.513691 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.513669    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sk4t9\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:26.515831 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.515812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-extensions-socket-volume\") pod \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\" (UID: \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\") "
Jul 28 14:32:26.515926 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.515857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55vlz\" (UniqueName: \"kubernetes.io/projected/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-kube-api-access-55vlz\") pod \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\" (UID: \"c366c25c-6859-4bf5-b2f8-7fa7a3077b7b\") "
Jul 28 14:32:26.515989 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.515954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srrhb\" (UniqueName: \"kubernetes.io/projected/ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2-kube-api-access-srrhb\") pod \"limitador-operator-controller-manager-859d7b78bd-j7psw\" (UID: \"ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"
Jul 28 14:32:26.516331 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.516305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" (UID: "c366c25c-6859-4bf5-b2f8-7fa7a3077b7b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:32:26.517884 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.517865    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:26.518132 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.518112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-kube-api-access-55vlz" (OuterVolumeSpecName: "kube-api-access-55vlz") pod "c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" (UID: "c366c25c-6859-4bf5-b2f8-7fa7a3077b7b"). InnerVolumeSpecName "kube-api-access-55vlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:32:26.520154 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.520127    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sk4t9\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:26.522353 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.522294    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:26.523728 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.523701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:26.524245 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:26.524215    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:26.617002 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.616964    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctvvx\" (UniqueName: \"kubernetes.io/projected/6cdb6b34-c212-4425-90db-e8aee678b1d5-kube-api-access-ctvvx\") pod \"6cdb6b34-c212-4425-90db-e8aee678b1d5\" (UID: \"6cdb6b34-c212-4425-90db-e8aee678b1d5\") "
Jul 28 14:32:26.617214 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.617111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srrhb\" (UniqueName: \"kubernetes.io/projected/ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2-kube-api-access-srrhb\") pod \"limitador-operator-controller-manager-859d7b78bd-j7psw\" (UID: \"ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"
Jul 28 14:32:26.617288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.617233    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55vlz\" (UniqueName: \"kubernetes.io/projected/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-kube-api-access-55vlz\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:26.617288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.617253    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b-extensions-socket-volume\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:26.619497 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.619473    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdb6b34-c212-4425-90db-e8aee678b1d5-kube-api-access-ctvvx" (OuterVolumeSpecName: "kube-api-access-ctvvx") pod "6cdb6b34-c212-4425-90db-e8aee678b1d5" (UID: "6cdb6b34-c212-4425-90db-e8aee678b1d5"). InnerVolumeSpecName "kube-api-access-ctvvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:32:26.624124 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.624096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:26.627617 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.627590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srrhb\" (UniqueName: \"kubernetes.io/projected/ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2-kube-api-access-srrhb\") pod \"limitador-operator-controller-manager-859d7b78bd-j7psw\" (UID: \"ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"
Jul 28 14:32:26.658925 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.658888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:26.665722 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.665699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:26.671818 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.671784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"
Jul 28 14:32:26.720385 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.718609    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctvvx\" (UniqueName: \"kubernetes.io/projected/6cdb6b34-c212-4425-90db-e8aee678b1d5-kube-api-access-ctvvx\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:26.825672 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:26.825647    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"]
Jul 28 14:32:26.829202 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:26.829173    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a5535b_a9f6_462a_b15a_22c3a224a7cd.slice/crio-b66f94d1a76d69ebb27bc5684bc3e57c964bb21d19d80b17cde4fefcdc3400aa WatchSource:0}: Error finding container b66f94d1a76d69ebb27bc5684bc3e57c964bb21d19d80b17cde4fefcdc3400aa: Status 404 returned error can't find the container with id b66f94d1a76d69ebb27bc5684bc3e57c964bb21d19d80b17cde4fefcdc3400aa
Jul 28 14:32:27.049958 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.049914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"]
Jul 28 14:32:27.054126 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:27.054084    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac04e0bc_ec1f_4e99_b31c_9eb9f875aae2.slice/crio-f658a300685059a1ac270836721a70ad75aff8ca47353a2f70a5de2b81d05525 WatchSource:0}: Error finding container f658a300685059a1ac270836721a70ad75aff8ca47353a2f70a5de2b81d05525: Status 404 returned error can't find the container with id f658a300685059a1ac270836721a70ad75aff8ca47353a2f70a5de2b81d05525
Jul 28 14:32:27.056208 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.056188    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"]
Jul 28 14:32:27.058933 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:27.058911    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402e90a4_c7d5_4db1_ad88_44315e785a10.slice/crio-4a05f57616fea44e0bae724c5c04f37fcdcb83168d57e6b48494c5bf0ce08983 WatchSource:0}: Error finding container 4a05f57616fea44e0bae724c5c04f37fcdcb83168d57e6b48494c5bf0ce08983: Status 404 returned error can't find the container with id 4a05f57616fea44e0bae724c5c04f37fcdcb83168d57e6b48494c5bf0ce08983
Jul 28 14:32:27.250622 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:27.250585    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zkzfl\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:27.252611 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:27.252572    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sk4t9\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:27.252833 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.252814    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cdb6b34-c212-4425-90db-e8aee678b1d5" path="/var/lib/kubelet/pods/6cdb6b34-c212-4425-90db-e8aee678b1d5/volumes"
Jul 28 14:32:27.253243 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.253223    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c366c25c-6859-4bf5-b2f8-7fa7a3077b7b" path="/var/lib/kubelet/pods/c366c25c-6859-4bf5-b2f8-7fa7a3077b7b/volumes"
Jul 28 14:32:27.254512 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:27.254483    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:27.474174 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.474079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw" event={"ID":"ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2","Type":"ContainerStarted","Data":"c1850b1eeb74bb2096208d920ec988e403eb1534808d01eacefa9a0950b1dfdb"}
Jul 28 14:32:27.474174 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.474131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw" event={"ID":"ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2","Type":"ContainerStarted","Data":"f658a300685059a1ac270836721a70ad75aff8ca47353a2f70a5de2b81d05525"}
Jul 28 14:32:27.474174 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.474166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"
Jul 28 14:32:27.475953 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.475919    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" containerName="manager" containerID="cri-o://f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914" gracePeriod=2
Jul 28 14:32:27.477560 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.477518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r" event={"ID":"402e90a4-c7d5-4db1-ad88-44315e785a10","Type":"ContainerStarted","Data":"c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472"}
Jul 28 14:32:27.477678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.477570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r" event={"ID":"402e90a4-c7d5-4db1-ad88-44315e785a10","Type":"ContainerStarted","Data":"4a05f57616fea44e0bae724c5c04f37fcdcb83168d57e6b48494c5bf0ce08983"}
Jul 28 14:32:27.477678 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.477591    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:27.479369 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.479343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t" event={"ID":"97a5535b-a9f6-462a-b15a-22c3a224a7cd","Type":"ContainerStarted","Data":"ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5"}
Jul 28 14:32:27.479480 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.479415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:27.479480 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.479438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t" event={"ID":"97a5535b-a9f6-462a-b15a-22c3a224a7cd","Type":"ContainerStarted","Data":"b66f94d1a76d69ebb27bc5684bc3e57c964bb21d19d80b17cde4fefcdc3400aa"}
Jul 28 14:32:27.480860 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.480841    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sk4t9"
Jul 28 14:32:27.480977 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.480887    2579 scope.go:117] "RemoveContainer" containerID="cbb9e8386439c4ef97a79b676732ef54d90ae2ac1ca2ef419ba9a4cd7f0d4962"
Jul 28 14:32:27.482338 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.482322    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zkzfl"
Jul 28 14:32:27.489805 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.489788    2579 scope.go:117] "RemoveContainer" containerID="357a92af67c1d9a99a4739f65c2bd0c67ad6831aa086c49fc3b5d7cadf204724"
Jul 28 14:32:27.496669 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.496625    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw" podStartSLOduration=1.496607003 podStartE2EDuration="1.496607003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:27.495714733 +0000 UTC m=+590.942564043" watchObservedRunningTime="2026-07-28 14:32:27.496607003 +0000 UTC m=+590.943456313"
Jul 28 14:32:27.525722 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.525660    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t" podStartSLOduration=1.525636414 podStartE2EDuration="1.525636414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:27.523938954 +0000 UTC m=+590.970788258" watchObservedRunningTime="2026-07-28 14:32:27.525636414 +0000 UTC m=+590.972485725"
Jul 28 14:32:27.549752 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.549696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r" podStartSLOduration=1.549680485 podStartE2EDuration="1.549680485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:27.54932598 +0000 UTC m=+590.996175280" watchObservedRunningTime="2026-07-28 14:32:27.549680485 +0000 UTC m=+590.996529793"
Jul 28 14:32:27.715877 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.715848    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:27.717989 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:27.717959    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:27.727494 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.727381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgr22\" (UniqueName: \"kubernetes.io/projected/731882f6-6766-40eb-b24d-c91fbffbf668-kube-api-access-xgr22\") pod \"731882f6-6766-40eb-b24d-c91fbffbf668\" (UID: \"731882f6-6766-40eb-b24d-c91fbffbf668\") "
Jul 28 14:32:27.727494 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.727479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/731882f6-6766-40eb-b24d-c91fbffbf668-extensions-socket-volume\") pod \"731882f6-6766-40eb-b24d-c91fbffbf668\" (UID: \"731882f6-6766-40eb-b24d-c91fbffbf668\") "
Jul 28 14:32:27.728035 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.728002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731882f6-6766-40eb-b24d-c91fbffbf668-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "731882f6-6766-40eb-b24d-c91fbffbf668" (UID: "731882f6-6766-40eb-b24d-c91fbffbf668"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:32:27.729685 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.729654    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731882f6-6766-40eb-b24d-c91fbffbf668-kube-api-access-xgr22" (OuterVolumeSpecName: "kube-api-access-xgr22") pod "731882f6-6766-40eb-b24d-c91fbffbf668" (UID: "731882f6-6766-40eb-b24d-c91fbffbf668"). InnerVolumeSpecName "kube-api-access-xgr22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:32:27.829033 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.828988    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgr22\" (UniqueName: \"kubernetes.io/projected/731882f6-6766-40eb-b24d-c91fbffbf668-kube-api-access-xgr22\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:27.829033 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:27.829027    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/731882f6-6766-40eb-b24d-c91fbffbf668-extensions-socket-volume\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:28.488736 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:28.488697    2579 generic.go:356] "Generic (PLEG): container finished" podID="731882f6-6766-40eb-b24d-c91fbffbf668" containerID="f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914" exitCode=0
Jul 28 14:32:28.488736 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:28.488733    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:28.489258 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:28.488792    2579 scope.go:117] "RemoveContainer" containerID="f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914"
Jul 28 14:32:28.491253 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:28.491226    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:28.498801 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:28.498784    2579 scope.go:117] "RemoveContainer" containerID="f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914"
Jul 28 14:32:28.499093 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:28.499064    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914\": container with ID starting with f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914 not found: ID does not exist" containerID="f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914"
Jul 28 14:32:28.499185 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:28.499104    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914"} err="failed to get container status \"f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914\": rpc error: code = NotFound desc = could not find container \"f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914\": container with ID starting with f39aedbc1ca65e8650b6639950e24a6b6eb418517e69dc1597a8b92ab367a914 not found: ID does not exist"
Jul 28 14:32:28.499776 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:28.499753    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cw98p\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cw98p"
Jul 28 14:32:28.972537 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:28.972501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:29.080342 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:29.080303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:29.251182 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:29.251093    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="731882f6-6766-40eb-b24d-c91fbffbf668" path="/var/lib/kubelet/pods/731882f6-6766-40eb-b24d-c91fbffbf668/volumes"
Jul 28 14:32:31.804771 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:31.804733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:31.923573 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:31.923538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:35.304497 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:35.304456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:35.491540 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:35.491503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:35.589921 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:35.589818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:38.492361 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.492326    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j7psw"
Jul 28 14:32:38.492798 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.492423    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:38.492798 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.492473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:38.546433 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.546378    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"]
Jul 28 14:32:38.546663 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.546638    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r" podUID="402e90a4-c7d5-4db1-ad88-44315e785a10" containerName="manager" containerID="cri-o://c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472" gracePeriod=10
Jul 28 14:32:38.799531 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.799500    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:38.849088 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.849044    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"]
Jul 28 14:32:38.849462 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.849443    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402e90a4-c7d5-4db1-ad88-44315e785a10" containerName="manager"
Jul 28 14:32:38.849462 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.849463    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="402e90a4-c7d5-4db1-ad88-44315e785a10" containerName="manager"
Jul 28 14:32:38.849648 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.849532    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="402e90a4-c7d5-4db1-ad88-44315e785a10" containerName="manager"
Jul 28 14:32:38.851312 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.851287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:38.868159 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.868125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"]
Jul 28 14:32:38.914405 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.914360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/402e90a4-c7d5-4db1-ad88-44315e785a10-extensions-socket-volume\") pod \"402e90a4-c7d5-4db1-ad88-44315e785a10\" (UID: \"402e90a4-c7d5-4db1-ad88-44315e785a10\") "
Jul 28 14:32:38.914587 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.914484    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcpcz\" (UniqueName: \"kubernetes.io/projected/402e90a4-c7d5-4db1-ad88-44315e785a10-kube-api-access-rcpcz\") pod \"402e90a4-c7d5-4db1-ad88-44315e785a10\" (UID: \"402e90a4-c7d5-4db1-ad88-44315e785a10\") "
Jul 28 14:32:38.914647 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.914589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drb95\" (UniqueName: \"kubernetes.io/projected/2044332f-6450-477a-b0dd-6ffec7cc8b85-kube-api-access-drb95\") pod \"kuadrant-operator-controller-manager-86878c769f-gw4z7\" (UID: \"2044332f-6450-477a-b0dd-6ffec7cc8b85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:38.914724 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.914686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044332f-6450-477a-b0dd-6ffec7cc8b85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gw4z7\" (UID: \"2044332f-6450-477a-b0dd-6ffec7cc8b85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:38.914817 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.914788    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e90a4-c7d5-4db1-ad88-44315e785a10-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "402e90a4-c7d5-4db1-ad88-44315e785a10" (UID: "402e90a4-c7d5-4db1-ad88-44315e785a10"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:32:38.916648 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:38.916624    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402e90a4-c7d5-4db1-ad88-44315e785a10-kube-api-access-rcpcz" (OuterVolumeSpecName: "kube-api-access-rcpcz") pod "402e90a4-c7d5-4db1-ad88-44315e785a10" (UID: "402e90a4-c7d5-4db1-ad88-44315e785a10"). InnerVolumeSpecName "kube-api-access-rcpcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:32:39.016125 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.016081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drb95\" (UniqueName: \"kubernetes.io/projected/2044332f-6450-477a-b0dd-6ffec7cc8b85-kube-api-access-drb95\") pod \"kuadrant-operator-controller-manager-86878c769f-gw4z7\" (UID: \"2044332f-6450-477a-b0dd-6ffec7cc8b85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:39.016313 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.016152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044332f-6450-477a-b0dd-6ffec7cc8b85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gw4z7\" (UID: \"2044332f-6450-477a-b0dd-6ffec7cc8b85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:39.016313 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.016183    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/402e90a4-c7d5-4db1-ad88-44315e785a10-extensions-socket-volume\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:39.016313 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.016193    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcpcz\" (UniqueName: \"kubernetes.io/projected/402e90a4-c7d5-4db1-ad88-44315e785a10-kube-api-access-rcpcz\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:39.016553 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.016534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044332f-6450-477a-b0dd-6ffec7cc8b85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gw4z7\" (UID: \"2044332f-6450-477a-b0dd-6ffec7cc8b85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:39.029869 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.029825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drb95\" (UniqueName: \"kubernetes.io/projected/2044332f-6450-477a-b0dd-6ffec7cc8b85-kube-api-access-drb95\") pod \"kuadrant-operator-controller-manager-86878c769f-gw4z7\" (UID: \"2044332f-6450-477a-b0dd-6ffec7cc8b85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:39.161155 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.161106    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:39.304914 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.304873    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"]
Jul 28 14:32:39.308062 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:39.308035    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2044332f_6450_477a_b0dd_6ffec7cc8b85.slice/crio-3bda01c15616b50d54434c1cd2c72bb0a14669797d4a8ba61d4aaaca49796992 WatchSource:0}: Error finding container 3bda01c15616b50d54434c1cd2c72bb0a14669797d4a8ba61d4aaaca49796992: Status 404 returned error can't find the container with id 3bda01c15616b50d54434c1cd2c72bb0a14669797d4a8ba61d4aaaca49796992
Jul 28 14:32:39.546086 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.546036    2579 generic.go:356] "Generic (PLEG): container finished" podID="402e90a4-c7d5-4db1-ad88-44315e785a10" containerID="c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472" exitCode=0
Jul 28 14:32:39.546086 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.546083    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"
Jul 28 14:32:39.546765 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.546122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r" event={"ID":"402e90a4-c7d5-4db1-ad88-44315e785a10","Type":"ContainerDied","Data":"c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472"}
Jul 28 14:32:39.546765 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.546175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r" event={"ID":"402e90a4-c7d5-4db1-ad88-44315e785a10","Type":"ContainerDied","Data":"4a05f57616fea44e0bae724c5c04f37fcdcb83168d57e6b48494c5bf0ce08983"}
Jul 28 14:32:39.546765 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.546207    2579 scope.go:117] "RemoveContainer" containerID="c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472"
Jul 28 14:32:39.549021 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.548992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7" event={"ID":"2044332f-6450-477a-b0dd-6ffec7cc8b85","Type":"ContainerStarted","Data":"0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f"}
Jul 28 14:32:39.549162 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.549050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7" event={"ID":"2044332f-6450-477a-b0dd-6ffec7cc8b85","Type":"ContainerStarted","Data":"3bda01c15616b50d54434c1cd2c72bb0a14669797d4a8ba61d4aaaca49796992"}
Jul 28 14:32:39.555454 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.555435    2579 scope.go:117] "RemoveContainer" containerID="c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472"
Jul 28 14:32:39.555773 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:39.555753    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472\": container with ID starting with c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472 not found: ID does not exist" containerID="c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472"
Jul 28 14:32:39.555835 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.555780    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472"} err="failed to get container status \"c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472\": rpc error: code = NotFound desc = could not find container \"c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472\": container with ID starting with c8497b213142b9b6ef67de02499c31f8d480225ade4e792f551d70b60a125472 not found: ID does not exist"
Jul 28 14:32:39.572591 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.572538    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7" podStartSLOduration=1.5725179740000002 podStartE2EDuration="1.572517974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:39.570843133 +0000 UTC m=+603.017692454" watchObservedRunningTime="2026-07-28 14:32:39.572517974 +0000 UTC m=+603.019367282"
Jul 28 14:32:39.591874 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.591830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"]
Jul 28 14:32:39.599950 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:39.599903    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6ck2r"]
Jul 28 14:32:40.556527 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:40.556492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:41.252499 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:41.252463    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="402e90a4-c7d5-4db1-ad88-44315e785a10" path="/var/lib/kubelet/pods/402e90a4-c7d5-4db1-ad88-44315e785a10/volumes"
Jul 28 14:32:44.249851 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:44.249814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:44.349548 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:44.349504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:47.278094 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:47.278057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:47.382793 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:47.382758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:47.478908 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:47.478878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:51.071312 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:51.071270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:51.173037 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:51.173002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:51.563037 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:51.563007    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:32:51.612790 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:51.612754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"]
Jul 28 14:32:51.612992 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:51.612966    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t" podUID="97a5535b-a9f6-462a-b15a-22c3a224a7cd" containerName="manager" containerID="cri-o://ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5" gracePeriod=10
Jul 28 14:32:51.886288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:51.886260    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:52.022269 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.022229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cpsv\" (UniqueName: \"kubernetes.io/projected/97a5535b-a9f6-462a-b15a-22c3a224a7cd-kube-api-access-5cpsv\") pod \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\" (UID: \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\") "
Jul 28 14:32:52.022269 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.022272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97a5535b-a9f6-462a-b15a-22c3a224a7cd-extensions-socket-volume\") pod \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\" (UID: \"97a5535b-a9f6-462a-b15a-22c3a224a7cd\") "
Jul 28 14:32:52.022770 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.022733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a5535b-a9f6-462a-b15a-22c3a224a7cd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "97a5535b-a9f6-462a-b15a-22c3a224a7cd" (UID: "97a5535b-a9f6-462a-b15a-22c3a224a7cd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:32:52.024571 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.024538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a5535b-a9f6-462a-b15a-22c3a224a7cd-kube-api-access-5cpsv" (OuterVolumeSpecName: "kube-api-access-5cpsv") pod "97a5535b-a9f6-462a-b15a-22c3a224a7cd" (UID: "97a5535b-a9f6-462a-b15a-22c3a224a7cd"). InnerVolumeSpecName "kube-api-access-5cpsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:32:52.123504 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.123365    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpsv\" (UniqueName: \"kubernetes.io/projected/97a5535b-a9f6-462a-b15a-22c3a224a7cd-kube-api-access-5cpsv\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:52.123504 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.123437    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97a5535b-a9f6-462a-b15a-22c3a224a7cd-extensions-socket-volume\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:32:52.610499 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.610460    2579 generic.go:356] "Generic (PLEG): container finished" podID="97a5535b-a9f6-462a-b15a-22c3a224a7cd" containerID="ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5" exitCode=0
Jul 28 14:32:52.610728 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.610514    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"
Jul 28 14:32:52.610728 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.610546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t" event={"ID":"97a5535b-a9f6-462a-b15a-22c3a224a7cd","Type":"ContainerDied","Data":"ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5"}
Jul 28 14:32:52.610728 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.610602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t" event={"ID":"97a5535b-a9f6-462a-b15a-22c3a224a7cd","Type":"ContainerDied","Data":"b66f94d1a76d69ebb27bc5684bc3e57c964bb21d19d80b17cde4fefcdc3400aa"}
Jul 28 14:32:52.610728 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.610628    2579 scope.go:117] "RemoveContainer" containerID="ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5"
Jul 28 14:32:52.619737 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.619583    2579 scope.go:117] "RemoveContainer" containerID="ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5"
Jul 28 14:32:52.619929 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:32:52.619909    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5\": container with ID starting with ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5 not found: ID does not exist" containerID="ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5"
Jul 28 14:32:52.619968 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.619938    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5"} err="failed to get container status \"ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5\": rpc error: code = NotFound desc = could not find container \"ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5\": container with ID starting with ce5a6162d1d9f54e981df976934d37b27b7603df39d6fb227811352c789294e5 not found: ID does not exist"
Jul 28 14:32:52.636983 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.636941    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"]
Jul 28 14:32:52.644294 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:52.644258    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l996t"]
Jul 28 14:32:53.252321 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:53.252281    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a5535b-a9f6-462a-b15a-22c3a224a7cd" path="/var/lib/kubelet/pods/97a5535b-a9f6-462a-b15a-22c3a224a7cd/volumes"
Jul 28 14:32:54.870180 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.870145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:54.965712 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.965670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:54.966875 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.966848    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"]
Jul 28 14:32:54.967341 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.967324    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a5535b-a9f6-462a-b15a-22c3a224a7cd" containerName="manager"
Jul 28 14:32:54.967341 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.967342    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a5535b-a9f6-462a-b15a-22c3a224a7cd" containerName="manager"
Jul 28 14:32:54.967523 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.967477    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a5535b-a9f6-462a-b15a-22c3a224a7cd" containerName="manager"
Jul 28 14:32:54.970151 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.970129    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:54.972954 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.972933    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2z627\""
Jul 28 14:32:54.973084 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.973068    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 14:32:54.983018 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:54.982989    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"]
Jul 28 14:32:55.039758 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.039707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:55.047844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.047803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ls6w\" (UniqueName: \"kubernetes.io/projected/9e868fe7-3c75-486d-8632-9b2840bdbf17-kube-api-access-6ls6w\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.047982 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.047852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e868fe7-3c75-486d-8632-9b2840bdbf17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.047982 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.047887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.047982 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.047914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.047982 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.047976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.048176 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.047996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.048176 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.048014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.048176 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.048067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.048176 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.048096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.141745 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.141644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:32:55.148735 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.148871 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.148871 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.148871 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.148871 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.148871 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.149094 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ls6w\" (UniqueName: \"kubernetes.io/projected/9e868fe7-3c75-486d-8632-9b2840bdbf17-kube-api-access-6ls6w\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.149094 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e868fe7-3c75-486d-8632-9b2840bdbf17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.149094 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.148954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.149235 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.149129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.149235 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.149175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.149342 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.149313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.149477 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.149452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.149811 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.149793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e868fe7-3c75-486d-8632-9b2840bdbf17-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.151315 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.151294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.151643 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.151623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.159677 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.159649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e868fe7-3c75-486d-8632-9b2840bdbf17-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.160164 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.160139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ls6w\" (UniqueName: \"kubernetes.io/projected/9e868fe7-3c75-486d-8632-9b2840bdbf17-kube-api-access-6ls6w\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gxhkr\" (UID: \"9e868fe7-3c75-486d-8632-9b2840bdbf17\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.280997 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.280958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:32:55.432737 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.432709    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"]
Jul 28 14:32:55.435177 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:32:55.435148    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e868fe7_3c75_486d_8632_9b2840bdbf17.slice/crio-f5ac6c667005a50a8d2c18c209688385dcd7df46bab0552c545691918963419d WatchSource:0}: Error finding container f5ac6c667005a50a8d2c18c209688385dcd7df46bab0552c545691918963419d: Status 404 returned error can't find the container with id f5ac6c667005a50a8d2c18c209688385dcd7df46bab0552c545691918963419d
Jul 28 14:32:55.625810 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:55.625771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" event={"ID":"9e868fe7-3c75-486d-8632-9b2840bdbf17","Type":"ContainerStarted","Data":"f5ac6c667005a50a8d2c18c209688385dcd7df46bab0552c545691918963419d"}
Jul 28 14:32:58.585886 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:58.585845    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:32:58.586189 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:58.585949    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:32:58.586189 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:58.585985    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:32:59.648914 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:59.648815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" event={"ID":"9e868fe7-3c75-486d-8632-9b2840bdbf17","Type":"ContainerStarted","Data":"1b5b4bafeeb06abacea568933af591f3c4731b060f88b0e7c4e6984ff425c7e5"}
Jul 28 14:32:59.672568 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:32:59.672504    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" podStartSLOduration=2.524123359 podStartE2EDuration="5.672486706s" totalImagesPullingTime="3.148363347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:32:59.671372365 +0000 UTC m=+623.118221676" watchObservedRunningTime="2026-07-28 14:32:59.672486706 +0000 UTC m=+623.119336015"
Jul 28 14:33:00.281666 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:00.281618    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:33:00.286779 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:00.286740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:33:00.653797 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:00.653761    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:33:00.654897 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:00.654873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:33:02.663546 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:02.663506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:02.766833 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:02.766795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:02.865617 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:02.865577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:05.320190 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:05.320152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:05.375583 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:05.375548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:11.675682 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:11.675644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:11.787070 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:11.787029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:11.843472 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:11.843438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:12.046168 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.046138    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:33:12.049258 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.049230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:12.051797 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.051774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7hsxh\""
Jul 28 14:33:12.051943 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.051772    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 14:33:12.060852 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.060822    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:33:12.073507 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.073468    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:33:12.196177 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.196135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2ec37ec5-3979-45d3-8cf6-83c9ba26ff96-config-file\") pod \"limitador-limitador-54d6d988d5-dlp89\" (UID: \"2ec37ec5-3979-45d3-8cf6-83c9ba26ff96\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:12.196360 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.196248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pw7c\" (UniqueName: \"kubernetes.io/projected/2ec37ec5-3979-45d3-8cf6-83c9ba26ff96-kube-api-access-9pw7c\") pod \"limitador-limitador-54d6d988d5-dlp89\" (UID: \"2ec37ec5-3979-45d3-8cf6-83c9ba26ff96\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:12.297621 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.297514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2ec37ec5-3979-45d3-8cf6-83c9ba26ff96-config-file\") pod \"limitador-limitador-54d6d988d5-dlp89\" (UID: \"2ec37ec5-3979-45d3-8cf6-83c9ba26ff96\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:12.297786 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.297627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pw7c\" (UniqueName: \"kubernetes.io/projected/2ec37ec5-3979-45d3-8cf6-83c9ba26ff96-kube-api-access-9pw7c\") pod \"limitador-limitador-54d6d988d5-dlp89\" (UID: \"2ec37ec5-3979-45d3-8cf6-83c9ba26ff96\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:12.298300 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.298280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2ec37ec5-3979-45d3-8cf6-83c9ba26ff96-config-file\") pod \"limitador-limitador-54d6d988d5-dlp89\" (UID: \"2ec37ec5-3979-45d3-8cf6-83c9ba26ff96\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:12.305340 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.305311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pw7c\" (UniqueName: \"kubernetes.io/projected/2ec37ec5-3979-45d3-8cf6-83c9ba26ff96-kube-api-access-9pw7c\") pod \"limitador-limitador-54d6d988d5-dlp89\" (UID: \"2ec37ec5-3979-45d3-8cf6-83c9ba26ff96\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:12.360857 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.360775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:12.532206 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.532124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:33:12.535133 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:33:12.535105    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec37ec5_3979_45d3_8cf6_83c9ba26ff96.slice/crio-8c8c8c0bc41386a32e0e3e5af2edfa029f6da4010ce87e48da1c18a2e3403512 WatchSource:0}: Error finding container 8c8c8c0bc41386a32e0e3e5af2edfa029f6da4010ce87e48da1c18a2e3403512: Status 404 returned error can't find the container with id 8c8c8c0bc41386a32e0e3e5af2edfa029f6da4010ce87e48da1c18a2e3403512
Jul 28 14:33:12.611261 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.611172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:12.689731 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.689691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:12.710442 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:12.710379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89" event={"ID":"2ec37ec5-3979-45d3-8cf6-83c9ba26ff96","Type":"ContainerStarted","Data":"8c8c8c0bc41386a32e0e3e5af2edfa029f6da4010ce87e48da1c18a2e3403512"}
Jul 28 14:33:15.725885 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:15.725836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89" event={"ID":"2ec37ec5-3979-45d3-8cf6-83c9ba26ff96","Type":"ContainerStarted","Data":"a6c21ea58b57527ff78f88aaa58b1b9aeca9f45086902cdc8cf32d95b2c1ff7a"}
Jul 28 14:33:15.726421 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:15.725913    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:15.743163 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:15.743104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89" podStartSLOduration=1.124426343 podStartE2EDuration="3.743084139s" totalImagesPullingTime="2.618657796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:33:15.741261698 +0000 UTC m=+639.188111008" watchObservedRunningTime="2026-07-28 14:33:15.743084139 +0000 UTC m=+639.189933448"
Jul 28 14:33:19.790882 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:19.790833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:19.846209 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:19.846176    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:26.737783 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:26.737750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dlp89"
Jul 28 14:33:32.248598 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:32.248555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:32.351183 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:32.351145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:32.761969 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:32.761926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:44.922788 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:44.922751    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:45.137010 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.136968    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pl584"]
Jul 28 14:33:45.148733 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.148701    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pl584"]
Jul 28 14:33:45.148915 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.148804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pl584"
Jul 28 14:33:45.151076 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.151048    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z5jp2\""
Jul 28 14:33:45.189603 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.189502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:45.264165 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.264128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzzb\" (UniqueName: \"kubernetes.io/projected/42a04c11-87ab-4bb2-87ed-a796b01eb71f-kube-api-access-jmzzb\") pod \"authorino-5d4b67c446-pl584\" (UID: \"42a04c11-87ab-4bb2-87ed-a796b01eb71f\") " pod="kuadrant-system/authorino-5d4b67c446-pl584"
Jul 28 14:33:45.364841 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.364808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzzb\" (UniqueName: \"kubernetes.io/projected/42a04c11-87ab-4bb2-87ed-a796b01eb71f-kube-api-access-jmzzb\") pod \"authorino-5d4b67c446-pl584\" (UID: \"42a04c11-87ab-4bb2-87ed-a796b01eb71f\") " pod="kuadrant-system/authorino-5d4b67c446-pl584"
Jul 28 14:33:45.373527 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.373494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzzb\" (UniqueName: \"kubernetes.io/projected/42a04c11-87ab-4bb2-87ed-a796b01eb71f-kube-api-access-jmzzb\") pod \"authorino-5d4b67c446-pl584\" (UID: \"42a04c11-87ab-4bb2-87ed-a796b01eb71f\") " pod="kuadrant-system/authorino-5d4b67c446-pl584"
Jul 28 14:33:45.377377 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.377340    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pl584"]
Jul 28 14:33:45.377620 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.377606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pl584"
Jul 28 14:33:45.518117 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.518091    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pl584"]
Jul 28 14:33:45.520936 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:33:45.520906    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42a04c11_87ab_4bb2_87ed_a796b01eb71f.slice/crio-594ba3052c60490add3ec44904ab30a10c12ba47e774f6626f4d0478061e9e05 WatchSource:0}: Error finding container 594ba3052c60490add3ec44904ab30a10c12ba47e774f6626f4d0478061e9e05: Status 404 returned error can't find the container with id 594ba3052c60490add3ec44904ab30a10c12ba47e774f6626f4d0478061e9e05
Jul 28 14:33:45.866828 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:45.866787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pl584" event={"ID":"42a04c11-87ab-4bb2-87ed-a796b01eb71f","Type":"ContainerStarted","Data":"594ba3052c60490add3ec44904ab30a10c12ba47e774f6626f4d0478061e9e05"}
Jul 28 14:33:49.887785 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:49.887738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pl584" event={"ID":"42a04c11-87ab-4bb2-87ed-a796b01eb71f","Type":"ContainerStarted","Data":"e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9"}
Jul 28 14:33:49.887785 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:49.887751    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-pl584" podUID="42a04c11-87ab-4bb2-87ed-a796b01eb71f" containerName="authorino" containerID="cri-o://e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9" gracePeriod=30
Jul 28 14:33:49.903661 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:49.903602    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-pl584" podStartSLOduration=1.260287778 podStartE2EDuration="4.903584421s" totalImagesPullingTime="3.643296643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:33:49.901912542 +0000 UTC m=+673.348761861" watchObservedRunningTime="2026-07-28 14:33:49.903584421 +0000 UTC m=+673.350433731"
Jul 28 14:33:50.152141 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.152065    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pl584"
Jul 28 14:33:50.208904 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.208850    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmzzb\" (UniqueName: \"kubernetes.io/projected/42a04c11-87ab-4bb2-87ed-a796b01eb71f-kube-api-access-jmzzb\") pod \"42a04c11-87ab-4bb2-87ed-a796b01eb71f\" (UID: \"42a04c11-87ab-4bb2-87ed-a796b01eb71f\") "
Jul 28 14:33:50.211562 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.211511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a04c11-87ab-4bb2-87ed-a796b01eb71f-kube-api-access-jmzzb" (OuterVolumeSpecName: "kube-api-access-jmzzb") pod "42a04c11-87ab-4bb2-87ed-a796b01eb71f" (UID: "42a04c11-87ab-4bb2-87ed-a796b01eb71f"). InnerVolumeSpecName "kube-api-access-jmzzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:33:50.310005 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.309957    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmzzb\" (UniqueName: \"kubernetes.io/projected/42a04c11-87ab-4bb2-87ed-a796b01eb71f-kube-api-access-jmzzb\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:33:50.446110 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.446007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:50.499975 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.499941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:50.893565 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.893526    2579 generic.go:356] "Generic (PLEG): container finished" podID="42a04c11-87ab-4bb2-87ed-a796b01eb71f" containerID="e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9" exitCode=0
Jul 28 14:33:50.893565 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.893566    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pl584"
Jul 28 14:33:50.894111 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.893586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pl584" event={"ID":"42a04c11-87ab-4bb2-87ed-a796b01eb71f","Type":"ContainerDied","Data":"e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9"}
Jul 28 14:33:50.894111 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.893628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pl584" event={"ID":"42a04c11-87ab-4bb2-87ed-a796b01eb71f","Type":"ContainerDied","Data":"594ba3052c60490add3ec44904ab30a10c12ba47e774f6626f4d0478061e9e05"}
Jul 28 14:33:50.894111 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.893652    2579 scope.go:117] "RemoveContainer" containerID="e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9"
Jul 28 14:33:50.902751 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.902719    2579 scope.go:117] "RemoveContainer" containerID="e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9"
Jul 28 14:33:50.903095 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:33:50.903073    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9\": container with ID starting with e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9 not found: ID does not exist" containerID="e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9"
Jul 28 14:33:50.903179 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.903101    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9"} err="failed to get container status \"e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9\": rpc error: code = NotFound desc = could not find container \"e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9\": container with ID starting with e8a838960c6f740bb0200b5b022c3d8d1298fedb390147184a77a606c5b13ba9 not found: ID does not exist"
Jul 28 14:33:50.915593 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.915560    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pl584"]
Jul 28 14:33:50.919134 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:50.919107    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pl584"]
Jul 28 14:33:51.253416 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:51.253302    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a04c11-87ab-4bb2-87ed-a796b01eb71f" path="/var/lib/kubelet/pods/42a04c11-87ab-4bb2-87ed-a796b01eb71f/volumes"
Jul 28 14:33:51.407075 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:51.407025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:51.488229 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:51.488186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:51.540762 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:51.540650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:52.082376 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:52.082340    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:52.303648 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:52.303605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:55.647649 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:55.647606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:33:55.753958 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:33:55.753921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:34:25.506860 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:25.506819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:34:27.092011 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:27.091971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:34:28.641417 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:28.641356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:34:35.097166 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:35.097125    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e868fe7-3c75-486d-8632-9b2840bdbf17" containerID="1b5b4bafeeb06abacea568933af591f3c4731b060f88b0e7c4e6984ff425c7e5" exitCode=0
Jul 28 14:34:35.097638 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:35.097180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" event={"ID":"9e868fe7-3c75-486d-8632-9b2840bdbf17","Type":"ContainerDied","Data":"1b5b4bafeeb06abacea568933af591f3c4731b060f88b0e7c4e6984ff425c7e5"}
Jul 28 14:34:35.097638 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:35.097573    2579 scope.go:117] "RemoveContainer" containerID="1b5b4bafeeb06abacea568933af591f3c4731b060f88b0e7c4e6984ff425c7e5"
Jul 28 14:34:35.098549 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:35.098523    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:34:35.098603 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:35.098590    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:34:35.098638 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:35.098623    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 14:34:36.104804 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:36.104766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" event={"ID":"9e868fe7-3c75-486d-8632-9b2840bdbf17","Type":"ContainerStarted","Data":"679c5016b27c1c65d1d18a42c8dd7a20da10e69a3a977cdb9db93714dad2bbac"}
Jul 28 14:34:36.180058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:36.180019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:34:36.271474 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:36.271434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:34:36.281329 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:36.281302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:34:36.281620 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:36.281594    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gxhkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 28 14:34:36.281670 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:36.281651    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" podUID="9e868fe7-3c75-486d-8632-9b2840bdbf17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 28 14:34:37.282140 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:37.282104    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gxhkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 28 14:34:37.282552 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:37.282200    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" podUID="9e868fe7-3c75-486d-8632-9b2840bdbf17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 28 14:34:38.281998 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:38.281952    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gxhkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 28 14:34:38.282194 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:38.282010    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" podUID="9e868fe7-3c75-486d-8632-9b2840bdbf17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 28 14:34:39.281827 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:39.281786    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gxhkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 28 14:34:39.282014 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:39.281848    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" podUID="9e868fe7-3c75-486d-8632-9b2840bdbf17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 28 14:34:40.281860 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:40.281816    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gxhkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 28 14:34:40.282263 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:40.281883    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr" podUID="9e868fe7-3c75-486d-8632-9b2840bdbf17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 28 14:34:41.285574 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:41.285542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:34:41.286036 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:41.285826    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:34:41.286506 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:41.286488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gxhkr"
Jul 28 14:34:41.625468 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:41.625429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:34:42.076592 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:42.076551    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:34:46.486122 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:46.486075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:34:47.277362 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:47.277326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:34:56.210546 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:56.210506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:34:57.685381 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:34:57.685337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:35:05.172515 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.172476    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"]
Jul 28 14:35:05.173001 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.172903    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a04c11-87ab-4bb2-87ed-a796b01eb71f" containerName="authorino"
Jul 28 14:35:05.173001 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.172922    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a04c11-87ab-4bb2-87ed-a796b01eb71f" containerName="authorino"
Jul 28 14:35:05.173001 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.172976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a04c11-87ab-4bb2-87ed-a796b01eb71f" containerName="authorino"
Jul 28 14:35:05.174962 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.174939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.177194 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.177171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 14:35:05.178225 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.178204    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 14:35:05.178351 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.178226    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 28 14:35:05.178575 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.178556    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bnhwc\""
Jul 28 14:35:05.186369 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.186337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"]
Jul 28 14:35:05.208710 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.208678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:35:05.265682 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.265643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.265896 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.265694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59sfv\" (UniqueName: \"kubernetes.io/projected/d4d80999-e812-4174-935a-8e5ee6639102-kube-api-access-59sfv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.265896 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.265796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.265896 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.265830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.265896 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.265862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d80999-e812-4174-935a-8e5ee6639102-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.266033 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.265907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.366722 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.366684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.366722 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.366727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.367002 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.366754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d80999-e812-4174-935a-8e5ee6639102-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.367002 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.366788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.367002 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.366828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.367002 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.366874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sfv\" (UniqueName: \"kubernetes.io/projected/d4d80999-e812-4174-935a-8e5ee6639102-kube-api-access-59sfv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.367222 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.367166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.367302 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.367271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.367364 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.367318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.369547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.369520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d80999-e812-4174-935a-8e5ee6639102-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.369748 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.369730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d80999-e812-4174-935a-8e5ee6639102-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.375051 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.375017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sfv\" (UniqueName: \"kubernetes.io/projected/d4d80999-e812-4174-935a-8e5ee6639102-kube-api-access-59sfv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms\" (UID: \"d4d80999-e812-4174-935a-8e5ee6639102\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.486644 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.486541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:05.627373 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.627226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"]
Jul 28 14:35:05.630952 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:35:05.630908    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d80999_e812_4174_935a_8e5ee6639102.slice/crio-2a205bc9993a8b393c875c89e5ed30c994d4c5195c36360ed45329efcaddefce WatchSource:0}: Error finding container 2a205bc9993a8b393c875c89e5ed30c994d4c5195c36360ed45329efcaddefce: Status 404 returned error can't find the container with id 2a205bc9993a8b393c875c89e5ed30c994d4c5195c36360ed45329efcaddefce
Jul 28 14:35:05.632271 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:05.632245    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:35:06.250006 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:06.249955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms" event={"ID":"d4d80999-e812-4174-935a-8e5ee6639102","Type":"ContainerStarted","Data":"2a205bc9993a8b393c875c89e5ed30c994d4c5195c36360ed45329efcaddefce"}
Jul 28 14:35:07.257263 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:07.257221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms" event={"ID":"d4d80999-e812-4174-935a-8e5ee6639102","Type":"ContainerStarted","Data":"db2bfa9acc6369e3c8e0e3a3e56ceb5887b1acdf7ad4fce0feb87bf34154bb6e"}
Jul 28 14:35:07.283992 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:07.283926    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms" podStartSLOduration=1.192120757 podStartE2EDuration="2.283904515s" totalImagesPullingTime="1.091783758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:35:07.282501478 +0000 UTC m=+750.729350789" watchObservedRunningTime="2026-07-28 14:35:07.283904515 +0000 UTC m=+750.730753878"
Jul 28 14:35:07.286563 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:07.286529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:35:09.427710 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:09.427668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:35:12.398881 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:12.398837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:35:12.496988 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:12.496952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:35:15.487495 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:15.487456    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:15.487991 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:15.487527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:15.500283 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:15.500244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:16.311615 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:16.311584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms"
Jul 28 14:35:18.013212 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:18.013173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:35:20.775828 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:20.775786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:35:39.086599 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.086571    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"]
Jul 28 14:35:39.088796 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.088779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.090940 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.090920    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 28 14:35:39.098839 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.098813    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"]
Jul 28 14:35:39.150412 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.150369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.150517 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.150424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.150517 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.150482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a62f39-1150-4205-a3d2-b8bc079587c7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.150517 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.150513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.150680 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.150535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.150680 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.150571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjzw\" (UniqueName: \"kubernetes.io/projected/f6a62f39-1150-4205-a3d2-b8bc079587c7-kube-api-access-cmjzw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.250872 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.250846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.251003 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.250892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.251003 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.250940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjzw\" (UniqueName: \"kubernetes.io/projected/f6a62f39-1150-4205-a3d2-b8bc079587c7-kube-api-access-cmjzw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.251003 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.250978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.251266 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.251012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.251266 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.251059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a62f39-1150-4205-a3d2-b8bc079587c7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.251266 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.251209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.251448 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.251281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.251448 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.251412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.253441 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.253416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a62f39-1150-4205-a3d2-b8bc079587c7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.254039 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.254006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a62f39-1150-4205-a3d2-b8bc079587c7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.259169 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.259136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjzw\" (UniqueName: \"kubernetes.io/projected/f6a62f39-1150-4205-a3d2-b8bc079587c7-kube-api-access-cmjzw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m\" (UID: \"f6a62f39-1150-4205-a3d2-b8bc079587c7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.398193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.398126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:39.522694 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:39.522663    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"]
Jul 28 14:35:39.526277 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:35:39.526238    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a62f39_1150_4205_a3d2_b8bc079587c7.slice/crio-1b3aefe50d57f08e7ee142bccda395324a49b271c2676f173cbc391a6be5960a WatchSource:0}: Error finding container 1b3aefe50d57f08e7ee142bccda395324a49b271c2676f173cbc391a6be5960a: Status 404 returned error can't find the container with id 1b3aefe50d57f08e7ee142bccda395324a49b271c2676f173cbc391a6be5960a
Jul 28 14:35:40.406839 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:40.406799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m" event={"ID":"f6a62f39-1150-4205-a3d2-b8bc079587c7","Type":"ContainerStarted","Data":"1b3aefe50d57f08e7ee142bccda395324a49b271c2676f173cbc391a6be5960a"}
Jul 28 14:35:40.477106 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:40.477071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:35:41.389157 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:41.389121    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:35:46.434548 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:46.434462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m" event={"ID":"f6a62f39-1150-4205-a3d2-b8bc079587c7","Type":"ContainerStarted","Data":"b0743875c125adf3d76bcc0db50a5cc81782e5f081ad114ade809dfcdd96af0c"}
Jul 28 14:35:46.450912 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:46.450869    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m" podStartSLOduration=0.866570879 podStartE2EDuration="7.450853735s" totalImagesPullingTime="6.584282856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:35:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:35:46.450073556 +0000 UTC m=+789.896922865" watchObservedRunningTime="2026-07-28 14:35:46.450853735 +0000 UTC m=+789.897703048"
Jul 28 14:35:49.398614 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:49.398573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:49.398614 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:49.398627    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:49.410643 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:49.410623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:35:49.456790 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:35:49.456764    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m"
Jul 28 14:36:38.078771 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:38.078736    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:36:53.560804 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.560774    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-545c77995f-nfg54"]
Jul 28 14:36:53.564129 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.564109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-545c77995f-nfg54"
Jul 28 14:36:53.567243 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.567223    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 14:36:53.567243 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.567235    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z5jp2\""
Jul 28 14:36:53.569840 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.569821    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-545c77995f-nfg54"]
Jul 28 14:36:53.575818 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.575795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgcv\" (UniqueName: \"kubernetes.io/projected/40a8465c-258b-4f2d-803b-d9526ae00c95-kube-api-access-wsgcv\") pod \"authorino-545c77995f-nfg54\" (UID: \"40a8465c-258b-4f2d-803b-d9526ae00c95\") " pod="kuadrant-system/authorino-545c77995f-nfg54"
Jul 28 14:36:53.575912 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.575895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a8465c-258b-4f2d-803b-d9526ae00c95-tls-cert\") pod \"authorino-545c77995f-nfg54\" (UID: \"40a8465c-258b-4f2d-803b-d9526ae00c95\") " pod="kuadrant-system/authorino-545c77995f-nfg54"
Jul 28 14:36:53.676581 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.676556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a8465c-258b-4f2d-803b-d9526ae00c95-tls-cert\") pod \"authorino-545c77995f-nfg54\" (UID: \"40a8465c-258b-4f2d-803b-d9526ae00c95\") " pod="kuadrant-system/authorino-545c77995f-nfg54"
Jul 28 14:36:53.676691 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.676605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgcv\" (UniqueName: \"kubernetes.io/projected/40a8465c-258b-4f2d-803b-d9526ae00c95-kube-api-access-wsgcv\") pod \"authorino-545c77995f-nfg54\" (UID: \"40a8465c-258b-4f2d-803b-d9526ae00c95\") " pod="kuadrant-system/authorino-545c77995f-nfg54"
Jul 28 14:36:53.678909 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.678887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a8465c-258b-4f2d-803b-d9526ae00c95-tls-cert\") pod \"authorino-545c77995f-nfg54\" (UID: \"40a8465c-258b-4f2d-803b-d9526ae00c95\") " pod="kuadrant-system/authorino-545c77995f-nfg54"
Jul 28 14:36:53.684074 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.684051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgcv\" (UniqueName: \"kubernetes.io/projected/40a8465c-258b-4f2d-803b-d9526ae00c95-kube-api-access-wsgcv\") pod \"authorino-545c77995f-nfg54\" (UID: \"40a8465c-258b-4f2d-803b-d9526ae00c95\") " pod="kuadrant-system/authorino-545c77995f-nfg54"
Jul 28 14:36:53.874700 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:53.874649    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-545c77995f-nfg54"
Jul 28 14:36:54.001634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:54.001609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-545c77995f-nfg54"]
Jul 28 14:36:54.004035 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:36:54.004007    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40a8465c_258b_4f2d_803b_d9526ae00c95.slice/crio-156e213eefacbb942353705e5e341cd0840a334c39556870af1f0896c3ebeaca WatchSource:0}: Error finding container 156e213eefacbb942353705e5e341cd0840a334c39556870af1f0896c3ebeaca: Status 404 returned error can't find the container with id 156e213eefacbb942353705e5e341cd0840a334c39556870af1f0896c3ebeaca
Jul 28 14:36:54.736795 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:54.736759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-545c77995f-nfg54" event={"ID":"40a8465c-258b-4f2d-803b-d9526ae00c95","Type":"ContainerStarted","Data":"156e213eefacbb942353705e5e341cd0840a334c39556870af1f0896c3ebeaca"}
Jul 28 14:36:56.748566 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:56.748531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-545c77995f-nfg54" event={"ID":"40a8465c-258b-4f2d-803b-d9526ae00c95","Type":"ContainerStarted","Data":"94ea17f4003f7a1b63c84225ef390a059558dd6a8d8fbadac683fbaec267f3ba"}
Jul 28 14:36:56.764505 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:56.764451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-545c77995f-nfg54" podStartSLOduration=1.822862469 podStartE2EDuration="3.764433965s" totalImagesPullingTime="1.941571496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:36:56.763769977 +0000 UTC m=+860.210619363" watchObservedRunningTime="2026-07-28 14:36:56.764433965 +0000 UTC m=+860.211283273"
Jul 28 14:36:57.303953 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:57.303911    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:36:57.403775 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:57.403742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:36:58.439504 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:58.439453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:36:58.537681 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:36:58.537647    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:37:26.887429 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:37:26.887385    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:37:36.475193 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:37:36.475159    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:37:47.785652 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:37:47.785592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:37:57.406278 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:37:57.406236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:38:20.704867 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:38:20.704829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:03.535901 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:03.535786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:04.758632 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:04.758594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:31.581862 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:31.581819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:39:40.588863 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:40.588823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:39:47.823256 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:47.823208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:49.174097 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:49.174060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:54.197645 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:54.197601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:54.262889 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:54.262837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:54.357290 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:54.357250    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:54.452547 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:54.452444    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:39:54.561273 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:39:54.561224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:40:00.644185 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:00.644130    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:40:00.819022 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:00.818979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:40:18.827730 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:18.827692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:40:18.923947 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:18.923900    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:40:19.478108 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:19.478065    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:40:27.485520 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:27.485467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:40:39.686317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:39.686271    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:40:47.084761 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:47.084716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:40:58.180031 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:40:58.179980    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:41:09.684098 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:41:09.684050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:41:20.781177 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:41:20.781132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:41:31.388227 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:41:31.388184    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:41:56.085328 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:41:56.085287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:42:00.383899 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:42:00.383808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:42:08.981712 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:42:08.981671    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:42:32.382377 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:42:32.382337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:42:46.390047 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:42:46.390001    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:42:57.182570 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:42:57.182529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:43:09.067767 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:43:09.067722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:43:11.692886 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:43:11.692840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:43:30.977818 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:43:30.977714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:43:43.183844 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:43:43.183795    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:43:53.082728 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:43:53.082695    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:44:17.490038 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:17.490003    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"]
Jul 28 14:44:17.490615 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:17.490195    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7" podUID="2044332f-6450-477a-b0dd-6ffec7cc8b85" containerName="manager" containerID="cri-o://0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f" gracePeriod=10
Jul 28 14:44:21.148058 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.148029    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:44:21.261778 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.261709    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044332f-6450-477a-b0dd-6ffec7cc8b85-extensions-socket-volume\") pod \"2044332f-6450-477a-b0dd-6ffec7cc8b85\" (UID: \"2044332f-6450-477a-b0dd-6ffec7cc8b85\") "
Jul 28 14:44:21.261778 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.261768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drb95\" (UniqueName: \"kubernetes.io/projected/2044332f-6450-477a-b0dd-6ffec7cc8b85-kube-api-access-drb95\") pod \"2044332f-6450-477a-b0dd-6ffec7cc8b85\" (UID: \"2044332f-6450-477a-b0dd-6ffec7cc8b85\") "
Jul 28 14:44:21.262113 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.262091    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2044332f-6450-477a-b0dd-6ffec7cc8b85-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2044332f-6450-477a-b0dd-6ffec7cc8b85" (UID: "2044332f-6450-477a-b0dd-6ffec7cc8b85"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:44:21.263711 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.263684    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2044332f-6450-477a-b0dd-6ffec7cc8b85-kube-api-access-drb95" (OuterVolumeSpecName: "kube-api-access-drb95") pod "2044332f-6450-477a-b0dd-6ffec7cc8b85" (UID: "2044332f-6450-477a-b0dd-6ffec7cc8b85"). InnerVolumeSpecName "kube-api-access-drb95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:44:21.362262 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.362232    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drb95\" (UniqueName: \"kubernetes.io/projected/2044332f-6450-477a-b0dd-6ffec7cc8b85-kube-api-access-drb95\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:44:21.362262 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.362258    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2044332f-6450-477a-b0dd-6ffec7cc8b85-extensions-socket-volume\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:44:21.888848 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.888813    2579 generic.go:356] "Generic (PLEG): container finished" podID="2044332f-6450-477a-b0dd-6ffec7cc8b85" containerID="0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f" exitCode=0
Jul 28 14:44:21.889016 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.888855    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"
Jul 28 14:44:21.889016 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.888899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7" event={"ID":"2044332f-6450-477a-b0dd-6ffec7cc8b85","Type":"ContainerDied","Data":"0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f"}
Jul 28 14:44:21.889016 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.888949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7" event={"ID":"2044332f-6450-477a-b0dd-6ffec7cc8b85","Type":"ContainerDied","Data":"3bda01c15616b50d54434c1cd2c72bb0a14669797d4a8ba61d4aaaca49796992"}
Jul 28 14:44:21.889016 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.888973    2579 scope.go:117] "RemoveContainer" containerID="0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f"
Jul 28 14:44:21.898562 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.898545    2579 scope.go:117] "RemoveContainer" containerID="0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f"
Jul 28 14:44:21.898789 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:44:21.898769    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f\": container with ID starting with 0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f not found: ID does not exist" containerID="0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f"
Jul 28 14:44:21.898824 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.898798    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f"} err="failed to get container status \"0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f\": rpc error: code = NotFound desc = could not find container \"0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f\": container with ID starting with 0a6dff943aaa174fc0254d1d91df85b8053baa36c12d9bd528f9191e445d6b0f not found: ID does not exist"
Jul 28 14:44:21.909879 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.909858    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"]
Jul 28 14:44:21.913874 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:21.913852    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gw4z7"]
Jul 28 14:44:23.251362 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:44:23.251329    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2044332f-6450-477a-b0dd-6ffec7cc8b85" path="/var/lib/kubelet/pods/2044332f-6450-477a-b0dd-6ffec7cc8b85/volumes"
Jul 28 14:45:00.145121 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.145029    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q"]
Jul 28 14:45:00.147507 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.145534    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2044332f-6450-477a-b0dd-6ffec7cc8b85" containerName="manager"
Jul 28 14:45:00.147507 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.145553    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2044332f-6450-477a-b0dd-6ffec7cc8b85" containerName="manager"
Jul 28 14:45:00.147507 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.145624    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2044332f-6450-477a-b0dd-6ffec7cc8b85" containerName="manager"
Jul 28 14:45:00.148446 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.148428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q"
Jul 28 14:45:00.150797 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.150756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 14:45:00.153121 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.153102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q"]
Jul 28 14:45:00.159784 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.159763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q"
Jul 28 14:45:00.292073 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.292042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q"]
Jul 28 14:45:00.294586 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:45:00.294558    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6b7a40_97f7_4bd6_a604_ecb6e8e9ec5f.slice/crio-5043030074bc06cf360c75b7cdeda4d0b0ebc408910c4f2dae8d532952ee58c3 WatchSource:0}: Error finding container 5043030074bc06cf360c75b7cdeda4d0b0ebc408910c4f2dae8d532952ee58c3: Status 404 returned error can't find the container with id 5043030074bc06cf360c75b7cdeda4d0b0ebc408910c4f2dae8d532952ee58c3
Jul 28 14:45:00.296460 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:00.296440    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:45:01.066477 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:01.066430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q" event={"ID":"6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f","Type":"ContainerStarted","Data":"5043030074bc06cf360c75b7cdeda4d0b0ebc408910c4f2dae8d532952ee58c3"}
Jul 28 14:45:03.076640 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:03.076600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q" event={"ID":"6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f","Type":"ContainerStarted","Data":"0f3669e66c73798a7a968df641b43c5de0c097052b5fbf9a86268ed528a20093"}
Jul 28 14:45:03.092880 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:03.092828    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q" podStartSLOduration=1.066151526 podStartE2EDuration="3.092814525s" totalImagesPullingTime="2.026662999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:45:03.091003278 +0000 UTC m=+1346.537852604" watchObservedRunningTime="2026-07-28 14:45:03.092814525 +0000 UTC m=+1346.539663880"
Jul 28 14:45:04.082089 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:04.082055    2579 generic.go:356] "Generic (PLEG): container finished" podID="6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f" containerID="0f3669e66c73798a7a968df641b43c5de0c097052b5fbf9a86268ed528a20093" exitCode=0
Jul 28 14:45:04.082469 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:04.082152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q" event={"ID":"6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f","Type":"ContainerDied","Data":"0f3669e66c73798a7a968df641b43c5de0c097052b5fbf9a86268ed528a20093"}
Jul 28 14:45:05.218803 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:05.218780    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q"
Jul 28 14:45:06.093422 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:06.093379    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q"
Jul 28 14:45:06.093422 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:06.093385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-r8b6q" event={"ID":"6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f","Type":"ContainerDied","Data":"5043030074bc06cf360c75b7cdeda4d0b0ebc408910c4f2dae8d532952ee58c3"}
Jul 28 14:45:06.093640 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:06.093448    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5043030074bc06cf360c75b7cdeda4d0b0ebc408910c4f2dae8d532952ee58c3"
Jul 28 14:45:18.686166 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.686130    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"]
Jul 28 14:45:18.686571 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.686488    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f" containerName="cleanup"
Jul 28 14:45:18.686571 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.686500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f" containerName="cleanup"
Jul 28 14:45:18.686647 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.686588    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f" containerName="cleanup"
Jul 28 14:45:18.692699 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.692677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:18.695240 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.695221    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sr8xr\""
Jul 28 14:45:18.704505 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.704485    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"]
Jul 28 14:45:18.831533 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.831486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsqd9\" (UniqueName: \"kubernetes.io/projected/2ef284aa-5f29-41ae-b720-eea7dc6a6c8d-kube-api-access-dsqd9\") pod \"kuadrant-operator-controller-manager-86878c769f-q59lf\" (UID: \"2ef284aa-5f29-41ae-b720-eea7dc6a6c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:18.831533 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.831546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ef284aa-5f29-41ae-b720-eea7dc6a6c8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q59lf\" (UID: \"2ef284aa-5f29-41ae-b720-eea7dc6a6c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:18.932198 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.932148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsqd9\" (UniqueName: \"kubernetes.io/projected/2ef284aa-5f29-41ae-b720-eea7dc6a6c8d-kube-api-access-dsqd9\") pod \"kuadrant-operator-controller-manager-86878c769f-q59lf\" (UID: \"2ef284aa-5f29-41ae-b720-eea7dc6a6c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:18.932198 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.932211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ef284aa-5f29-41ae-b720-eea7dc6a6c8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q59lf\" (UID: \"2ef284aa-5f29-41ae-b720-eea7dc6a6c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:18.932615 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.932598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ef284aa-5f29-41ae-b720-eea7dc6a6c8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q59lf\" (UID: \"2ef284aa-5f29-41ae-b720-eea7dc6a6c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:18.939938 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:18.939893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsqd9\" (UniqueName: \"kubernetes.io/projected/2ef284aa-5f29-41ae-b720-eea7dc6a6c8d-kube-api-access-dsqd9\") pod \"kuadrant-operator-controller-manager-86878c769f-q59lf\" (UID: \"2ef284aa-5f29-41ae-b720-eea7dc6a6c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:19.003631 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:19.003596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:19.142508 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:19.142482    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"]
Jul 28 14:45:19.145034 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:45:19.145011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef284aa_5f29_41ae_b720_eea7dc6a6c8d.slice/crio-30b4575ad3a13d0c82d35c299458973c578b7fe739541e56fed6cccf2d0f95b3 WatchSource:0}: Error finding container 30b4575ad3a13d0c82d35c299458973c578b7fe739541e56fed6cccf2d0f95b3: Status 404 returned error can't find the container with id 30b4575ad3a13d0c82d35c299458973c578b7fe739541e56fed6cccf2d0f95b3
Jul 28 14:45:19.155101 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:19.155076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf" event={"ID":"2ef284aa-5f29-41ae-b720-eea7dc6a6c8d","Type":"ContainerStarted","Data":"30b4575ad3a13d0c82d35c299458973c578b7fe739541e56fed6cccf2d0f95b3"}
Jul 28 14:45:20.161650 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:20.161614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf" event={"ID":"2ef284aa-5f29-41ae-b720-eea7dc6a6c8d","Type":"ContainerStarted","Data":"1c7b6e2df2d539c8eb991e6def88222c43c56b174cf4dd92773cb2c60dbf9fda"}
Jul 28 14:45:20.162020 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:20.161771    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:20.179798 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:20.179739    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf" podStartSLOduration=2.179721492 podStartE2EDuration="2.179721492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:45:20.178158785 +0000 UTC m=+1363.625008094" watchObservedRunningTime="2026-07-28 14:45:20.179721492 +0000 UTC m=+1363.626570803"
Jul 28 14:45:31.168162 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:31.168134    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q59lf"
Jul 28 14:45:38.383651 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:38.383617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:45:50.275894 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:50.275855    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:45:57.252713 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:45:57.252671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:46:03.373542 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:03.373505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:46:03.473705 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:03.473656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:46:05.087649 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:05.087612    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:46:14.580775 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:14.580733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:46:29.079185 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:29.079150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:46:37.676344 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:37.676308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:46:48.380712 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:48.380652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:46:54.881819 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:54.881770    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:46:59.303560 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:46:59.303504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:47:07.798285 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:47:07.798243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:47:17.580183 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:47:17.580142    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:47:30.780975 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:47:30.780943    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:47:42.484736 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:47:42.484690    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:47:54.520159 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:47:54.520120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:48:04.784728 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:48:04.784641    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:48:06.823477 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:48:06.823427    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:48:19.978792 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:48:19.978744    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:48:24.880805 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:48:24.880765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:48:28.976531 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:48:28.976492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:48:55.745668 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:48:55.745623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:48:55.802313 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:48:55.802281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:48:58.672927 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:48:58.672887    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:49:12.770798 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:49:12.770765    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:49:23.086928 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:49:23.086888    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:49:35.079491 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:49:35.079457    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:49:46.780802 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:49:46.780762    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:49:55.577975 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:49:55.577941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:50:06.486963 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:50:06.486928    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:50:15.803249 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:50:15.803214    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:50:25.225411 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:50:25.225359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:50:38.786040 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:50:38.786001    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:50:53.576030 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:50:53.575988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:51:03.879955 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:51:03.879865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:51:10.580824 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:51:10.580772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:51:20.574502 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:51:20.574464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:51:27.074857 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:51:27.074820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:51:43.785062 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:51:43.785022    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:51:56.487616 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:51:56.487579    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:52:06.338590 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:06.338549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:10.583992 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:10.583953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:52:12.120068 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:12.120036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:17.814346 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:17.814303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:19.683299 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:19.683251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:52:20.858109 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:20.858051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:20.958531 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:20.958479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:38.192747 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:38.192710    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:41.318787 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:41.318745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:41.419254 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:41.419209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:41.793123 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:41.793013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:42.774977 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:42.774925    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:52:47.684304 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:47.684265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:47.784502 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:47.784462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:48.010452 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:48.010345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:48.280448 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:48.280348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:48.661907 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:48.661869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:53.133797 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:53.133754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:53.209462 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:53.209415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:53.280979 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:53.280939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:53.658832 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:53.658796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:53.836819 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:53.836778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:53.931202 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:53.931111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:52:56.581700 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:52:56.581667    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:53:33.937461 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:33.937375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:34.008126 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:34.008084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:34.074286 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:34.074247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:34.133707 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:34.133665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:34.268468 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:34.268346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:34.555634 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:34.555590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:34.796194 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:34.796142    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:34.892283 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:34.892190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:35.014701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:35.014667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:35.445620 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:35.445585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:35.542602 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:35.542562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:37.502632 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:37.502589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:37.609623 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:37.609590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:37.756147 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:37.756036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:38.084012 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:38.083971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:53:38.120968 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:38.120930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:38.210802 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:38.210757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:46.807366 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:46.807321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:46.854284 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:46.854242    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:53:48.376788 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:53:48.376749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:54:08.886141 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:08.886045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:08.953466 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:08.953425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:09.017723 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:09.017686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:09.076994 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:09.076957    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:09.203324 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:09.203231    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:09.454681 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:09.454589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:09.703137 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:09.703100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:10.026361 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:10.026318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:10.352513 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:10.352477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:11.076109 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:11.076074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:54:12.598969 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:12.598922    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:12.676996 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:12.676956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:12.746658 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:12.746625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:12.871357 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:12.871257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:13.274641 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:13.274534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:13.372288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:13.372238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:22.682604 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:22.682563    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:54:43.497215 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:43.497166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:43.566637 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:43.566604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:43.622339 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:43.622297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:43.690028 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:43.689997    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:45.046054 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:45.046022    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:45.291065 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:45.291027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:45.550777 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:45.550749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:45.863210 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:45.863120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:46.649920 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:46.649867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:47.376254 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:47.376220    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:54:47.818846 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:47.818803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:47.886780 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:47.886742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:47.955620 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:47.955564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:48.113686 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:48.113595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:48.524138 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:48.524053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:48.625841 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:48.625802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:54:58.775983 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:54:58.775939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:55:18.679454 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:18.679414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:18.779205 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:18.779170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:18.830677 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:18.830639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:18.972257 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:18.972162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:19.236501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:19.236414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:19.715299 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:19.715261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:19.812545 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:19.812512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:19.913404 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:19.913369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:21.370714 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:21.370673    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:22.782858 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:22.782820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:55:27.288903 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:27.288863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:27.396940 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:27.396893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:27.489832 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:27.489791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:27.589444 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:27.589384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:28.013991 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:28.013901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:55:34.782098 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:34.782063    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:55:54.004358 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:55:54.004316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:00.415362 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:00.415324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:00.519131 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:00.519094    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:00.633961 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:00.633920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:16.585464 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:16.585426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:22.948461 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:22.948420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:23.058531 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:23.058497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:23.153602 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:23.153560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:33.704247 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:33.704200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:34.557288 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:34.557245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:41.140232 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:41.140194    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:41.243072 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:41.243033    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:41.345631 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:41.345593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:41.519059 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:41.518969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:45.079701 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:45.079665    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:56:46.725219 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:46.725182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:46.815974 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:46.815936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:46.913857 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:46.913826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:48.896638 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:48.896610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:54.849626 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:54.849590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:54.914209 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:54.914176    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:56:55.089636 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:55.089599    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:56:55.606083 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:56:55.606045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:01.618858 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:01.618773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:01.680337 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:01.680302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:01.898317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:01.898230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:02.221504 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:02.221373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:02.321897 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:02.321852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:04.383221 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:04.383176    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:57:05.123339 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:05.123297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:05.203896 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:05.203844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:05.279185 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:05.279146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:05.516935 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:05.516828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:05.730158 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:05.730119    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:05.813510 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:05.813473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:08.284788 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:08.284745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:15.977522 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:15.977486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:57:20.262446 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:20.262386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:20.314301 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:20.314265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:30.196859 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:30.196819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:30.892069 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:30.892021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:30.995002 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:30.994964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:31.434257 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:31.434215    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:37.384693 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:37.384663    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:37.450668 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:37.450634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:48.176704 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:48.176668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:48.278717 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:48.278676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:48.338075 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:48.338039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:48.715673 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:48.715636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:50.976952 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:50.976916    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:57:54.432096 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:54.432061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:54.608092 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:54.608060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:54.717647 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:54.717552    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:54.948199 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:54.948160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:55.066563 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:55.066509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:55.165286 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:55.165248    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:57:55.292048 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:57:55.292013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:01.370842 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:01.370807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:01.477456 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:01.477417    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:01.570273 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:01.570240    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:01.743797 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:01.743703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:07.728142 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:07.728103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:07.781669 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:07.781629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:14.766047 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:14.766013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:15.115366 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:15.115328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:15.290722 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:15.290686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:18.484224 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:18.484177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:18.569716 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:18.569677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:18.633731 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:18.633687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:18.811635 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:18.811593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:19.251688 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:19.251584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:19.339818 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:19.339778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:24.777271 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:24.777232    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:58:43.593483 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:43.593446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:48.777881 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.777840    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"]
Jul 28 14:58:48.781748 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.781728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.784203 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.784179    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-96a65772\"/\"openshift-service-ca.crt\""
Jul 28 14:58:48.784334 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.784263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-96a65772\"/\"default-dockercfg-524cn\""
Jul 28 14:58:48.784988 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.784965    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-96a65772\"/\"e2e-lifecycle-model-96a65772-kserve-self-signed-certs\""
Jul 28 14:58:48.784988 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.784985    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-96a65772\"/\"kube-root-ca.crt\""
Jul 28 14:58:48.790362 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.790343    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"]
Jul 28 14:58:48.813181 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.813154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-model-cache\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.813317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.813191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-home\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.813317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.813216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jt7\" (UniqueName: \"kubernetes.io/projected/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-kube-api-access-m9jt7\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.813317 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.813303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-dshm\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.813470 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.813336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tmp-dir\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.813470 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.813369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tls-certs\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.825964 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.825932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:58:48.914210 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tmp-dir\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.914457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tls-certs\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.914457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-model-cache\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.914457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-home\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.914457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jt7\" (UniqueName: \"kubernetes.io/projected/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-kube-api-access-m9jt7\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.914457 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-dshm\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.914752 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tmp-dir\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.914752 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-home\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.914851 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.914770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-model-cache\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.916637 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.916614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-dshm\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.916921 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.916905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tls-certs\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:48.922570 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:48.922544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jt7\" (UniqueName: \"kubernetes.io/projected/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-kube-api-access-m9jt7\") pod \"e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") " pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:49.093508 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:49.093472    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:49.230612 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:49.230583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"]
Jul 28 14:58:49.232532 ip-10-0-141-160 kubenswrapper[2579]: W0728 14:58:49.232502    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod142fa8df_38fa_48a8_86e0_8cda1f3bbf41.slice/crio-c2c35761c4d19920cd34917b7dfc6cc8c5eb99f26bcbe96eee16d6ecd0f7699c WatchSource:0}: Error finding container c2c35761c4d19920cd34917b7dfc6cc8c5eb99f26bcbe96eee16d6ecd0f7699c: Status 404 returned error can't find the container with id c2c35761c4d19920cd34917b7dfc6cc8c5eb99f26bcbe96eee16d6ecd0f7699c
Jul 28 14:58:49.941472 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:49.941431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9" event={"ID":"142fa8df-38fa-48a8-86e0-8cda1f3bbf41","Type":"ContainerStarted","Data":"797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99"}
Jul 28 14:58:49.941855 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:49.941490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9" event={"ID":"142fa8df-38fa-48a8-86e0-8cda1f3bbf41","Type":"ContainerStarted","Data":"c2c35761c4d19920cd34917b7dfc6cc8c5eb99f26bcbe96eee16d6ecd0f7699c"}
Jul 28 14:58:49.957763 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:49.957712    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9" podStartSLOduration=1.957697113 podStartE2EDuration="1.957697113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:58:49.956053417 +0000 UTC m=+2173.402902728" watchObservedRunningTime="2026-07-28 14:58:49.957697113 +0000 UTC m=+2173.404546422"
Jul 28 14:58:52.282503 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:52.282468    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:58:59.094068 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:59.094024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:59.094068 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:59.094078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:59.106994 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:59.106964    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:58:59.998350 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:58:59.998317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:59:00.760324 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:00.760287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:08.477869 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:08.477831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:59:27.329402 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:27.329346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:59:27.939484 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:27.939450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:28.040165 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:28.040135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:28.890983 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:28.890949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:28.954441 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:28.954406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:29.049685 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:29.049654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:29.098751 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:29.098709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:32.735059 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:32.735007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:37.721312 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:37.721281    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"]
Jul 28 14:59:37.775582 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:37.775552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 14:59:37.778050 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:37.778025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 14:59:52.741187 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:52.741144    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9" podUID="142fa8df-38fa-48a8-86e0-8cda1f3bbf41" containerName="main" containerID="cri-o://797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99" gracePeriod=45
Jul 28 14:59:52.994124 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:52.994061    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:59:53.081847 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.081818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-model-cache\") pod \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") "
Jul 28 14:59:53.082012 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.081866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-home\") pod \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") "
Jul 28 14:59:53.082012 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.081888    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-dshm\") pod \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") "
Jul 28 14:59:53.082012 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.081910    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tls-certs\") pod \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") "
Jul 28 14:59:53.082012 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.081970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9jt7\" (UniqueName: \"kubernetes.io/projected/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-kube-api-access-m9jt7\") pod \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") "
Jul 28 14:59:53.082012 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.082003    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tmp-dir\") pod \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\" (UID: \"142fa8df-38fa-48a8-86e0-8cda1f3bbf41\") "
Jul 28 14:59:53.082291 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.082139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-model-cache" (OuterVolumeSpecName: "model-cache") pod "142fa8df-38fa-48a8-86e0-8cda1f3bbf41" (UID: "142fa8df-38fa-48a8-86e0-8cda1f3bbf41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:59:53.082291 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.082156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-home" (OuterVolumeSpecName: "home") pod "142fa8df-38fa-48a8-86e0-8cda1f3bbf41" (UID: "142fa8df-38fa-48a8-86e0-8cda1f3bbf41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:59:53.082425 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.082317    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "142fa8df-38fa-48a8-86e0-8cda1f3bbf41" (UID: "142fa8df-38fa-48a8-86e0-8cda1f3bbf41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:59:53.082425 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.082345    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:59:53.082425 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.082364    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:59:53.084099 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.084066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "142fa8df-38fa-48a8-86e0-8cda1f3bbf41" (UID: "142fa8df-38fa-48a8-86e0-8cda1f3bbf41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:59:53.084203 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.084109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-dshm" (OuterVolumeSpecName: "dshm") pod "142fa8df-38fa-48a8-86e0-8cda1f3bbf41" (UID: "142fa8df-38fa-48a8-86e0-8cda1f3bbf41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:59:53.084203 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.084115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-kube-api-access-m9jt7" (OuterVolumeSpecName: "kube-api-access-m9jt7") pod "142fa8df-38fa-48a8-86e0-8cda1f3bbf41" (UID: "142fa8df-38fa-48a8-86e0-8cda1f3bbf41"). InnerVolumeSpecName "kube-api-access-m9jt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:59:53.183717 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.183678    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9jt7\" (UniqueName: \"kubernetes.io/projected/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-kube-api-access-m9jt7\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:59:53.183717 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.183713    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:59:53.183717 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.183725    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:59:53.183954 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.183733    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142fa8df-38fa-48a8-86e0-8cda1f3bbf41-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 14:59:53.235501 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.235465    2579 generic.go:356] "Generic (PLEG): container finished" podID="142fa8df-38fa-48a8-86e0-8cda1f3bbf41" containerID="797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99" exitCode=0
Jul 28 14:59:53.235670 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.235514    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"
Jul 28 14:59:53.235670 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.235528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9" event={"ID":"142fa8df-38fa-48a8-86e0-8cda1f3bbf41","Type":"ContainerDied","Data":"797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99"}
Jul 28 14:59:53.235670 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.235576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9" event={"ID":"142fa8df-38fa-48a8-86e0-8cda1f3bbf41","Type":"ContainerDied","Data":"c2c35761c4d19920cd34917b7dfc6cc8c5eb99f26bcbe96eee16d6ecd0f7699c"}
Jul 28 14:59:53.235670 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.235594    2579 scope.go:117] "RemoveContainer" containerID="797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99"
Jul 28 14:59:53.244534 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.244519    2579 scope.go:117] "RemoveContainer" containerID="797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99"
Jul 28 14:59:53.244785 ip-10-0-141-160 kubenswrapper[2579]: E0728 14:59:53.244763    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99\": container with ID starting with 797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99 not found: ID does not exist" containerID="797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99"
Jul 28 14:59:53.244845 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.244800    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99"} err="failed to get container status \"797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99\": rpc error: code = NotFound desc = could not find container \"797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99\": container with ID starting with 797e68081173e0501fc9d56731ba5a4a1cd84fe927fef07f4c70d0c4c3f1ba99 not found: ID does not exist"
Jul 28 14:59:53.257914 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.257892    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"]
Jul 28 14:59:53.258907 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:53.258888    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-96a65772/e2e-lifecycle-model-96a65772-kserve-6cd4fbf47f-2ccm9"]
Jul 28 14:59:55.251214 ip-10-0-141-160 kubenswrapper[2579]: I0728 14:59:55.251177    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="142fa8df-38fa-48a8-86e0-8cda1f3bbf41" path="/var/lib/kubelet/pods/142fa8df-38fa-48a8-86e0-8cda1f3bbf41/volumes"
Jul 28 15:00:14.424664 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:14.424611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:15.060469 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:15.060427    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:20.605968 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:20.605925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:43.617949 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:43.617915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:43.670624 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:43.670595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:44.106312 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:44.106269    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:44.650624 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:44.650590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:46.993779 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:46.993726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:47.093380 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:47.093345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:47.248692 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:47.248583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:47.879614 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:47.879577    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:00:48.376232 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:48.376186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:48.478279 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:48.478234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:48.581252 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:48.581204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:50.825263 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:50.825219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:57.368834 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:57.368800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:57.466105 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:57.466072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:58.092719 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:58.092681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:00:58.473796 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:00:58.473719    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:01:03.834888 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:03.834839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:03.927414 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:03.927364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:04.013867 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:04.013828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:08.797519 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:08.797483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:08.895755 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:08.895715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:14.731707 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:14.731649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:15.181256 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:15.181214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:36.244412 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:36.244358    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:42.820549 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:42.820491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:42.922198 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:42.922157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:43.382972 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:43.382935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:43.589487 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:43.589445    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:44.262859 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:44.262823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:46.535700 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:46.535647    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:46.634330 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:46.634293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:46.806244 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:46.806205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:47.115290 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:47.115183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:47.261265 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:47.261225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:47.357178 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:47.357131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:01:53.281285 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:01:53.281251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:02:11.640337 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:11.640292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:02:13.362159 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:13.362114    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:02:15.718229 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:15.718193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:02:15.817216 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:15.817183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:02:16.723145 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:16.723103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:02:16.824212 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:16.824171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:02:22.974214 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:22.974174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:02:40.811986 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:40.811948    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:02:43.680327 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.680287    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"]
Jul 28 15:02:43.680902 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.680795    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="142fa8df-38fa-48a8-86e0-8cda1f3bbf41" containerName="main"
Jul 28 15:02:43.680902 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.680816    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="142fa8df-38fa-48a8-86e0-8cda1f3bbf41" containerName="main"
Jul 28 15:02:43.681032 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.680912    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="142fa8df-38fa-48a8-86e0-8cda1f3bbf41" containerName="main"
Jul 28 15:02:43.684201 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.684181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.686304 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.686281    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d392e9\"/\"openshift-service-ca.crt\""
Jul 28 15:02:43.686414 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.686355    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d392e9\"/\"kube-root-ca.crt\""
Jul 28 15:02:43.687242 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.687229    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d392e9\"/\"test-model-d392e9-kserve-self-signed-certs\""
Jul 28 15:02:43.687297 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.687259    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d392e9\"/\"default-dockercfg-zzk54\""
Jul 28 15:02:43.693876 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.693858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"]
Jul 28 15:02:43.706968 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.706942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:02:43.870721 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.870686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-dshm\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.870869 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.870737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxx7f\" (UniqueName: \"kubernetes.io/projected/e737eccf-3494-43de-b12c-803d95df2aee-kube-api-access-hxx7f\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.870869 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.870793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e737eccf-3494-43de-b12c-803d95df2aee-tls-certs\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.870869 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.870844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-tmp-dir\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.871016 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.870875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-model-cache\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.871079 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.871007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-home\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971495 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-tmp-dir\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971495 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-model-cache\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971698 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-home\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971698 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-dshm\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971698 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxx7f\" (UniqueName: \"kubernetes.io/projected/e737eccf-3494-43de-b12c-803d95df2aee-kube-api-access-hxx7f\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971698 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e737eccf-3494-43de-b12c-803d95df2aee-tls-certs\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971917 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-tmp-dir\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971917 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-model-cache\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.971995 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.971914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-home\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.973756 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.973736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-dshm\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.974029 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.974012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e737eccf-3494-43de-b12c-803d95df2aee-tls-certs\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.978531 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.978503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxx7f\" (UniqueName: \"kubernetes.io/projected/e737eccf-3494-43de-b12c-803d95df2aee-kube-api-access-hxx7f\") pod \"test-model-d392e9-kserve-75f987757-plrfc\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") " pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:43.994567 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:43.994539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:44.131118 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:02:44.131080    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode737eccf_3494_43de_b12c_803d95df2aee.slice/crio-5015fffe9cc05f31cc55d64fd8d4a3c08cf582b49fccb2c73d5c4a93fa42c47b WatchSource:0}: Error finding container 5015fffe9cc05f31cc55d64fd8d4a3c08cf582b49fccb2c73d5c4a93fa42c47b: Status 404 returned error can't find the container with id 5015fffe9cc05f31cc55d64fd8d4a3c08cf582b49fccb2c73d5c4a93fa42c47b
Jul 28 15:02:44.131385 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:44.131362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"]
Jul 28 15:02:45.048150 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:45.048115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc" event={"ID":"e737eccf-3494-43de-b12c-803d95df2aee","Type":"ContainerStarted","Data":"de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194"}
Jul 28 15:02:45.048150 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:45.048155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc" event={"ID":"e737eccf-3494-43de-b12c-803d95df2aee","Type":"ContainerStarted","Data":"5015fffe9cc05f31cc55d64fd8d4a3c08cf582b49fccb2c73d5c4a93fa42c47b"}
Jul 28 15:02:45.062649 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:45.062606    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc" podStartSLOduration=2.06259166 podStartE2EDuration="2.06259166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:02:45.061801691 +0000 UTC m=+2408.508651000" watchObservedRunningTime="2026-07-28 15:02:45.06259166 +0000 UTC m=+2408.509440968"
Jul 28 15:02:46.695254 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:46.695220    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:02:53.995403 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:53.995365    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:53.995826 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:53.995431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:54.007609 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:54.007581    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:54.099035 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:54.099011    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:02:58.567248 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:58.567208    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:02:58.862404 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:02:58.862298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:03:10.357323 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.357200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"]
Jul 28 15:03:10.360856 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.360835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.363996 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.363974    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-904baa\"/\"default-dockercfg-d7frv\""
Jul 28 15:03:10.364111 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.363985    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-904baa\"/\"openshift-service-ca.crt\""
Jul 28 15:03:10.364111 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.363987    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-904baa\"/\"test-model-904baa-kserve-self-signed-certs\""
Jul 28 15:03:10.364228 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.364036    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-904baa\"/\"kube-root-ca.crt\""
Jul 28 15:03:10.366407 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.366370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"]
Jul 28 15:03:10.377942 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.377917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:03:10.478914 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.478887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbba060-f007-4cac-9ff0-b0a718707830-tls-certs\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.479043 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.478936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-model-cache\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.479043 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.478961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-dshm\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.479043 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.478980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzpm\" (UniqueName: \"kubernetes.io/projected/1cbba060-f007-4cac-9ff0-b0a718707830-kube-api-access-2tzpm\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.479043 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.479001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-tmp-dir\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.479043 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.479029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-home\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580089 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-model-cache\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-dshm\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzpm\" (UniqueName: \"kubernetes.io/projected/1cbba060-f007-4cac-9ff0-b0a718707830-kube-api-access-2tzpm\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-tmp-dir\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-home\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbba060-f007-4cac-9ff0-b0a718707830-tls-certs\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580567 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-model-cache\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580615 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-tmp-dir\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.580661 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.580642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-home\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.582370 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.582345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-dshm\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.582673 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.582657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbba060-f007-4cac-9ff0-b0a718707830-tls-certs\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.587046 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.587025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzpm\" (UniqueName: \"kubernetes.io/projected/1cbba060-f007-4cac-9ff0-b0a718707830-kube-api-access-2tzpm\") pod \"test-model-904baa-kserve-854bffc6b7-vn4hv\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") " pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.673104 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.673037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:10.804311 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:10.804284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"]
Jul 28 15:03:10.806414 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:03:10.806366    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cbba060_f007_4cac_9ff0_b0a718707830.slice/crio-fa463692c4e820c90ff57150f88605e65f1b68276a51e971776c8197d86a706b WatchSource:0}: Error finding container fa463692c4e820c90ff57150f88605e65f1b68276a51e971776c8197d86a706b: Status 404 returned error can't find the container with id fa463692c4e820c90ff57150f88605e65f1b68276a51e971776c8197d86a706b
Jul 28 15:03:11.165098 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:11.165058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv" event={"ID":"1cbba060-f007-4cac-9ff0-b0a718707830","Type":"ContainerStarted","Data":"5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3"}
Jul 28 15:03:11.165098 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:11.165106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv" event={"ID":"1cbba060-f007-4cac-9ff0-b0a718707830","Type":"ContainerStarted","Data":"fa463692c4e820c90ff57150f88605e65f1b68276a51e971776c8197d86a706b"}
Jul 28 15:03:11.178957 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:11.178907    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv" podStartSLOduration=1.178893377 podStartE2EDuration="1.178893377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:03:11.177880873 +0000 UTC m=+2434.624730183" watchObservedRunningTime="2026-07-28 15:03:11.178893377 +0000 UTC m=+2434.625742686"
Jul 28 15:03:14.975604 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:14.975566    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:03:20.673245 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:20.673198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:20.673656 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:20.673262    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:20.685611 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:20.685586    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:21.223037 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:21.223005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:03:28.578361 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:28.578324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:03:50.500075 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:50.500040    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"]
Jul 28 15:03:50.520005 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:50.519974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:03:51.429449 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:51.429378    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"]
Jul 28 15:03:51.825233 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:51.825200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:03:54.880285 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:03:54.880247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:04:02.074468 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:02.074430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:02.153341 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:02.153308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:02.203142 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:02.203109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:02.561825 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:02.561787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:05.519206 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.519169    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc" podUID="e737eccf-3494-43de-b12c-803d95df2aee" containerName="main" containerID="cri-o://de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194" gracePeriod=45
Jul 28 15:04:05.771090 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.771030    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:04:05.841135 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e737eccf-3494-43de-b12c-803d95df2aee-tls-certs\") pod \"e737eccf-3494-43de-b12c-803d95df2aee\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") "
Jul 28 15:04:05.841309 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxx7f\" (UniqueName: \"kubernetes.io/projected/e737eccf-3494-43de-b12c-803d95df2aee-kube-api-access-hxx7f\") pod \"e737eccf-3494-43de-b12c-803d95df2aee\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") "
Jul 28 15:04:05.841309 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-home\") pod \"e737eccf-3494-43de-b12c-803d95df2aee\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") "
Jul 28 15:04:05.841309 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841223    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-dshm\") pod \"e737eccf-3494-43de-b12c-803d95df2aee\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") "
Jul 28 15:04:05.841309 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-tmp-dir\") pod \"e737eccf-3494-43de-b12c-803d95df2aee\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") "
Jul 28 15:04:05.841309 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-model-cache\") pod \"e737eccf-3494-43de-b12c-803d95df2aee\" (UID: \"e737eccf-3494-43de-b12c-803d95df2aee\") "
Jul 28 15:04:05.841619 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-home" (OuterVolumeSpecName: "home") pod "e737eccf-3494-43de-b12c-803d95df2aee" (UID: "e737eccf-3494-43de-b12c-803d95df2aee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:05.841619 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841524    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e737eccf-3494-43de-b12c-803d95df2aee" (UID: "e737eccf-3494-43de-b12c-803d95df2aee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:05.841619 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.841605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-model-cache" (OuterVolumeSpecName: "model-cache") pod "e737eccf-3494-43de-b12c-803d95df2aee" (UID: "e737eccf-3494-43de-b12c-803d95df2aee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:05.843544 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.843518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-dshm" (OuterVolumeSpecName: "dshm") pod "e737eccf-3494-43de-b12c-803d95df2aee" (UID: "e737eccf-3494-43de-b12c-803d95df2aee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:05.843658 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.843602    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e737eccf-3494-43de-b12c-803d95df2aee-kube-api-access-hxx7f" (OuterVolumeSpecName: "kube-api-access-hxx7f") pod "e737eccf-3494-43de-b12c-803d95df2aee" (UID: "e737eccf-3494-43de-b12c-803d95df2aee"). InnerVolumeSpecName "kube-api-access-hxx7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:05.843658 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.843607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e737eccf-3494-43de-b12c-803d95df2aee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e737eccf-3494-43de-b12c-803d95df2aee" (UID: "e737eccf-3494-43de-b12c-803d95df2aee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:04:05.942338 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.942305    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxx7f\" (UniqueName: \"kubernetes.io/projected/e737eccf-3494-43de-b12c-803d95df2aee-kube-api-access-hxx7f\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:05.942338 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.942332    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:05.942538 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.942346    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:05.942538 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.942358    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:05.942538 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.942371    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e737eccf-3494-43de-b12c-803d95df2aee-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:05.942538 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:05.942386    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e737eccf-3494-43de-b12c-803d95df2aee-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:06.414212 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.414175    2579 generic.go:356] "Generic (PLEG): container finished" podID="e737eccf-3494-43de-b12c-803d95df2aee" containerID="de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194" exitCode=0
Jul 28 15:04:06.414405 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.414235    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"
Jul 28 15:04:06.414405 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.414261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc" event={"ID":"e737eccf-3494-43de-b12c-803d95df2aee","Type":"ContainerDied","Data":"de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194"}
Jul 28 15:04:06.414405 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.414304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc" event={"ID":"e737eccf-3494-43de-b12c-803d95df2aee","Type":"ContainerDied","Data":"5015fffe9cc05f31cc55d64fd8d4a3c08cf582b49fccb2c73d5c4a93fa42c47b"}
Jul 28 15:04:06.414405 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.414321    2579 scope.go:117] "RemoveContainer" containerID="de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194"
Jul 28 15:04:06.424634 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.424606    2579 scope.go:117] "RemoveContainer" containerID="de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194"
Jul 28 15:04:06.424919 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:04:06.424898    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194\": container with ID starting with de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194 not found: ID does not exist" containerID="de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194"
Jul 28 15:04:06.424999 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.424926    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194"} err="failed to get container status \"de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194\": rpc error: code = NotFound desc = could not find container \"de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194\": container with ID starting with de1511bfc807242ee980a12fd70da48f74f460c091ad97f3b1c2ef9bd8897194 not found: ID does not exist"
Jul 28 15:04:06.434876 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.434850    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"]
Jul 28 15:04:06.436825 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.436804    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d392e9/test-model-d392e9-kserve-75f987757-plrfc"]
Jul 28 15:04:06.448631 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.448583    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv" podUID="1cbba060-f007-4cac-9ff0-b0a718707830" containerName="main" containerID="cri-o://5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3" gracePeriod=45
Jul 28 15:04:06.706356 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.706331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:04:06.748346 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-dshm\") pod \"1cbba060-f007-4cac-9ff0-b0a718707830\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") "
Jul 28 15:04:06.748489 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-home\") pod \"1cbba060-f007-4cac-9ff0-b0a718707830\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") "
Jul 28 15:04:06.748489 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748414    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tzpm\" (UniqueName: \"kubernetes.io/projected/1cbba060-f007-4cac-9ff0-b0a718707830-kube-api-access-2tzpm\") pod \"1cbba060-f007-4cac-9ff0-b0a718707830\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") "
Jul 28 15:04:06.748489 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-model-cache\") pod \"1cbba060-f007-4cac-9ff0-b0a718707830\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") "
Jul 28 15:04:06.748489 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-tmp-dir\") pod \"1cbba060-f007-4cac-9ff0-b0a718707830\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") "
Jul 28 15:04:06.748710 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748505    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbba060-f007-4cac-9ff0-b0a718707830-tls-certs\") pod \"1cbba060-f007-4cac-9ff0-b0a718707830\" (UID: \"1cbba060-f007-4cac-9ff0-b0a718707830\") "
Jul 28 15:04:06.748710 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748677    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-home" (OuterVolumeSpecName: "home") pod "1cbba060-f007-4cac-9ff0-b0a718707830" (UID: "1cbba060-f007-4cac-9ff0-b0a718707830"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:06.748819 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748705    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cbba060-f007-4cac-9ff0-b0a718707830" (UID: "1cbba060-f007-4cac-9ff0-b0a718707830"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:06.748819 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.748767    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cbba060-f007-4cac-9ff0-b0a718707830" (UID: "1cbba060-f007-4cac-9ff0-b0a718707830"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:06.750635 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.750610    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-dshm" (OuterVolumeSpecName: "dshm") pod "1cbba060-f007-4cac-9ff0-b0a718707830" (UID: "1cbba060-f007-4cac-9ff0-b0a718707830"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:04:06.750729 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.750653    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cbba060-f007-4cac-9ff0-b0a718707830-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cbba060-f007-4cac-9ff0-b0a718707830" (UID: "1cbba060-f007-4cac-9ff0-b0a718707830"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:04:06.750729 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.750666    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbba060-f007-4cac-9ff0-b0a718707830-kube-api-access-2tzpm" (OuterVolumeSpecName: "kube-api-access-2tzpm") pod "1cbba060-f007-4cac-9ff0-b0a718707830" (UID: "1cbba060-f007-4cac-9ff0-b0a718707830"). InnerVolumeSpecName "kube-api-access-2tzpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:04:06.849983 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.849955    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:06.849983 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.849979    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbba060-f007-4cac-9ff0-b0a718707830-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:06.849983 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.849987    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:06.850178 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.849994    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:06.850178 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.850011    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tzpm\" (UniqueName: \"kubernetes.io/projected/1cbba060-f007-4cac-9ff0-b0a718707830-kube-api-access-2tzpm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:06.850178 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:06.850020    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cbba060-f007-4cac-9ff0-b0a718707830-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:07.251775 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.251740    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e737eccf-3494-43de-b12c-803d95df2aee" path="/var/lib/kubelet/pods/e737eccf-3494-43de-b12c-803d95df2aee/volumes"
Jul 28 15:04:07.419660 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.419632    2579 generic.go:356] "Generic (PLEG): container finished" podID="1cbba060-f007-4cac-9ff0-b0a718707830" containerID="5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3" exitCode=0
Jul 28 15:04:07.419800 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.419678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"
Jul 28 15:04:07.419800 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.419723    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv" event={"ID":"1cbba060-f007-4cac-9ff0-b0a718707830","Type":"ContainerDied","Data":"5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3"}
Jul 28 15:04:07.419800 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.419763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv" event={"ID":"1cbba060-f007-4cac-9ff0-b0a718707830","Type":"ContainerDied","Data":"fa463692c4e820c90ff57150f88605e65f1b68276a51e971776c8197d86a706b"}
Jul 28 15:04:07.419800 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.419787    2579 scope.go:117] "RemoveContainer" containerID="5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3"
Jul 28 15:04:07.427882 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.427861    2579 scope.go:117] "RemoveContainer" containerID="5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3"
Jul 28 15:04:07.428124 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:04:07.428098    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3\": container with ID starting with 5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3 not found: ID does not exist" containerID="5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3"
Jul 28 15:04:07.428192 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.428122    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3"} err="failed to get container status \"5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3\": rpc error: code = NotFound desc = could not find container \"5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3\": container with ID starting with 5e65efc409c6bb61aa8fa402ee7e595ed5278d8583f438c61c3ebf48b63a91a3 not found: ID does not exist"
Jul 28 15:04:07.435313 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.435292    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"]
Jul 28 15:04:07.437685 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:07.437663    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-904baa/test-model-904baa-kserve-854bffc6b7-vn4hv"]
Jul 28 15:04:08.478090 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:08.478057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:08.544091 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:08.544058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:08.753855 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:08.753780    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:08.996839 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:08.996807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:09.124472 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.124442    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"]
Jul 28 15:04:09.124903 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.124887    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e737eccf-3494-43de-b12c-803d95df2aee" containerName="main"
Jul 28 15:04:09.124944 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.124906    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e737eccf-3494-43de-b12c-803d95df2aee" containerName="main"
Jul 28 15:04:09.124944 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.124934    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cbba060-f007-4cac-9ff0-b0a718707830" containerName="main"
Jul 28 15:04:09.125008 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.124943    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cbba060-f007-4cac-9ff0-b0a718707830" containerName="main"
Jul 28 15:04:09.125048 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.125026    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e737eccf-3494-43de-b12c-803d95df2aee" containerName="main"
Jul 28 15:04:09.125048 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.125040    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cbba060-f007-4cac-9ff0-b0a718707830" containerName="main"
Jul 28 15:04:09.129334 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.129317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"
Jul 28 15:04:09.131419 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.131366    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 15:04:09.132132 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.132035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:04:09.133601 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.133578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"]
Jul 28 15:04:09.168941 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.168916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb07a541-f1f8-46e5-94c7-69d98af3cc7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-904baa-85b8k\" (UID: \"cb07a541-f1f8-46e5-94c7-69d98af3cc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"
Jul 28 15:04:09.252348 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.252321    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cbba060-f007-4cac-9ff0-b0a718707830" path="/var/lib/kubelet/pods/1cbba060-f007-4cac-9ff0-b0a718707830/volumes"
Jul 28 15:04:09.269756 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.269724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb07a541-f1f8-46e5-94c7-69d98af3cc7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-904baa-85b8k\" (UID: \"cb07a541-f1f8-46e5-94c7-69d98af3cc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"
Jul 28 15:04:09.270604 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.270583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb07a541-f1f8-46e5-94c7-69d98af3cc7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-904baa-85b8k\" (UID: \"cb07a541-f1f8-46e5-94c7-69d98af3cc7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"
Jul 28 15:04:09.441641 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.441556    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"
Jul 28 15:04:09.572661 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:09.572632    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"]
Jul 28 15:04:09.575282 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:04:09.575254    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb07a541_f1f8_46e5_94c7_69d98af3cc7b.slice/crio-ef1d03dd46826db28879ead72b22c9460544953360c5e9124558f0766db1c272 WatchSource:0}: Error finding container ef1d03dd46826db28879ead72b22c9460544953360c5e9124558f0766db1c272: Status 404 returned error can't find the container with id ef1d03dd46826db28879ead72b22c9460544953360c5e9124558f0766db1c272
Jul 28 15:04:10.437063 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:10.437034    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb07a541-f1f8-46e5-94c7-69d98af3cc7b" containerID="747c3a78f83dcbcf6afa774bc6a6ce1d2ea553859c8d8c1d9571ba4f8098e86a" exitCode=0
Jul 28 15:04:10.437214 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:10.437128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k" event={"ID":"cb07a541-f1f8-46e5-94c7-69d98af3cc7b","Type":"ContainerDied","Data":"747c3a78f83dcbcf6afa774bc6a6ce1d2ea553859c8d8c1d9571ba4f8098e86a"}
Jul 28 15:04:10.437214 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:10.437179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k" event={"ID":"cb07a541-f1f8-46e5-94c7-69d98af3cc7b","Type":"ContainerStarted","Data":"ef1d03dd46826db28879ead72b22c9460544953360c5e9124558f0766db1c272"}
Jul 28 15:04:11.079585 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:11.079554    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:04:11.573278 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:11.573256    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"
Jul 28 15:04:11.693943 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:11.693866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb07a541-f1f8-46e5-94c7-69d98af3cc7b-maas-api-service-ca\") pod \"cb07a541-f1f8-46e5-94c7-69d98af3cc7b\" (UID: \"cb07a541-f1f8-46e5-94c7-69d98af3cc7b\") "
Jul 28 15:04:11.694220 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:11.694194    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb07a541-f1f8-46e5-94c7-69d98af3cc7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb07a541-f1f8-46e5-94c7-69d98af3cc7b" (UID: "cb07a541-f1f8-46e5-94c7-69d98af3cc7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:04:11.795267 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:11.795230    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb07a541-f1f8-46e5-94c7-69d98af3cc7b-maas-api-service-ca\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:04:12.447036 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:12.446961    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"
Jul 28 15:04:12.447424 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:12.446960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k" event={"ID":"cb07a541-f1f8-46e5-94c7-69d98af3cc7b","Type":"ContainerDied","Data":"ef1d03dd46826db28879ead72b22c9460544953360c5e9124558f0766db1c272"}
Jul 28 15:04:12.447424 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:12.447085    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef1d03dd46826db28879ead72b22c9460544953360c5e9124558f0766db1c272"
Jul 28 15:04:19.659857 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:19.659812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:19.760246 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:19.760213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:19.830829 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:19.830799    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"]
Jul 28 15:04:19.833893 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:19.833841    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:19.834208 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:19.834186    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-904baa-85b8k"]
Jul 28 15:04:19.889770 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:19.889736    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:20.263236 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:20.263202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:21.251581 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:21.251550    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb07a541-f1f8-46e5-94c7-69d98af3cc7b" path="/var/lib/kubelet/pods/cb07a541-f1f8-46e5-94c7-69d98af3cc7b/volumes"
Jul 28 15:04:22.970818 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:22.970783    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:04:26.141059 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:26.141025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:26.239664 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:26.239633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:26.416236 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:26.416157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:26.659664 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:26.659627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:27.024486 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:27.024446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:27.448380 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:27.448342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:27.506128 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:27.506084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:30.342106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:30.342011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:30.429798 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:30.429768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:30.493151 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:30.493120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:30.922367 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:30.922328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:31.016706 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:31.016667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:35.075966 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:35.075929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:04:55.442468 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:55.442428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:55.952478 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:55.952447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:56.286796 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:56.286702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:56.384680 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:56.384644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:59.533077 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:59.533032    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:59.611784 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:59.611755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:04:59.677547 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:04:59.677515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:00.147595 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:00.147555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:00.267704 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:00.267653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:00.338954 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:00.338915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:03.577446 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:03.577416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:05:24.640647 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:24.640598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:28.008424 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.008373    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"]
Jul 28 15:05:28.008870 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.008767    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb07a541-f1f8-46e5-94c7-69d98af3cc7b" containerName="revoke-keys"
Jul 28 15:05:28.008870 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.008782    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb07a541-f1f8-46e5-94c7-69d98af3cc7b" containerName="revoke-keys"
Jul 28 15:05:28.008870 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.008853    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb07a541-f1f8-46e5-94c7-69d98af3cc7b" containerName="revoke-keys"
Jul 28 15:05:28.011819 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.011801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"
Jul 28 15:05:28.013833 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.013803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 15:05:28.013970 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.013839    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:05:28.017549 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.017521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"]
Jul 28 15:05:28.070471 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.070448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95eadd2f-2965-4392-82a2-bde46505d39b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l\" (UID: \"95eadd2f-2965-4392-82a2-bde46505d39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"
Jul 28 15:05:28.170759 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.170734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95eadd2f-2965-4392-82a2-bde46505d39b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l\" (UID: \"95eadd2f-2965-4392-82a2-bde46505d39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"
Jul 28 15:05:28.171314 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.171295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95eadd2f-2965-4392-82a2-bde46505d39b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l\" (UID: \"95eadd2f-2965-4392-82a2-bde46505d39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"
Jul 28 15:05:28.325369 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.325344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"
Jul 28 15:05:28.450878 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.450831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"]
Jul 28 15:05:28.454677 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:05:28.454650    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95eadd2f_2965_4392_82a2_bde46505d39b.slice/crio-c341dc4e047f8be3a553ad5201e08eb12d07fac088a743c9c2940664b25bd285 WatchSource:0}: Error finding container c341dc4e047f8be3a553ad5201e08eb12d07fac088a743c9c2940664b25bd285: Status 404 returned error can't find the container with id c341dc4e047f8be3a553ad5201e08eb12d07fac088a743c9c2940664b25bd285
Jul 28 15:05:28.798521 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.798484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l" event={"ID":"95eadd2f-2965-4392-82a2-bde46505d39b","Type":"ContainerStarted","Data":"3f3714180696ef40c2561eb7f0cd4a710b1d135ac013ed75074669b9f39a1aa1"}
Jul 28 15:05:28.798622 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.798539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l" event={"ID":"95eadd2f-2965-4392-82a2-bde46505d39b","Type":"ContainerStarted","Data":"c341dc4e047f8be3a553ad5201e08eb12d07fac088a743c9c2940664b25bd285"}
Jul 28 15:05:28.812003 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:28.811960    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l" podStartSLOduration=1.811943276 podStartE2EDuration="1.811943276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:05:28.811038016 +0000 UTC m=+2572.257887327" watchObservedRunningTime="2026-07-28 15:05:28.811943276 +0000 UTC m=+2572.258792584"
Jul 28 15:05:29.804124 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:29.804092    2579 generic.go:356] "Generic (PLEG): container finished" podID="95eadd2f-2965-4392-82a2-bde46505d39b" containerID="3f3714180696ef40c2561eb7f0cd4a710b1d135ac013ed75074669b9f39a1aa1" exitCode=0
Jul 28 15:05:29.804499 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:29.804185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l" event={"ID":"95eadd2f-2965-4392-82a2-bde46505d39b","Type":"ContainerDied","Data":"3f3714180696ef40c2561eb7f0cd4a710b1d135ac013ed75074669b9f39a1aa1"}
Jul 28 15:05:30.937886 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:30.937864    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"
Jul 28 15:05:30.992666 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:30.992641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95eadd2f-2965-4392-82a2-bde46505d39b-maas-api-service-ca\") pod \"95eadd2f-2965-4392-82a2-bde46505d39b\" (UID: \"95eadd2f-2965-4392-82a2-bde46505d39b\") "
Jul 28 15:05:30.992918 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:30.992897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95eadd2f-2965-4392-82a2-bde46505d39b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95eadd2f-2965-4392-82a2-bde46505d39b" (UID: "95eadd2f-2965-4392-82a2-bde46505d39b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:05:31.093501 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:31.093445    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95eadd2f-2965-4392-82a2-bde46505d39b-maas-api-service-ca\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:05:31.814190 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:31.814158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l" event={"ID":"95eadd2f-2965-4392-82a2-bde46505d39b","Type":"ContainerDied","Data":"c341dc4e047f8be3a553ad5201e08eb12d07fac088a743c9c2940664b25bd285"}
Jul 28 15:05:31.814190 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:31.814181    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"
Jul 28 15:05:31.814426 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:31.814204    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c341dc4e047f8be3a553ad5201e08eb12d07fac088a743c9c2940664b25bd285"
Jul 28 15:05:39.157175 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:39.157124    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:43.580540 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:43.580505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:43.651424 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:43.651368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:43.708041 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:43.708009    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:43.736476 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:43.736449    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"]
Jul 28 15:05:43.738944 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:43.738923    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7d80d5-hjg7l"]
Jul 28 15:05:43.761787 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:43.761753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:44.181134 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:44.181093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:45.251818 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:45.251789    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95eadd2f-2965-4392-82a2-bde46505d39b" path="/var/lib/kubelet/pods/95eadd2f-2965-4392-82a2-bde46505d39b/volumes"
Jul 28 15:05:49.973645 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:49.973611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:05:50.284812 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:50.284736    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:50.384815 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:50.384788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:50.585767 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:50.585724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:05:50.932011 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:05:50.931926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:06.587437 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:06.587334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:06.686712 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:06.686676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:06.742467 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:06.742432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:07.163758 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:07.163723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:10.872643 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:10.872608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:06:13.104076 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:13.104035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:13.204777 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:13.204746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:13.409023 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:13.408929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:13.671425 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:13.671315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:14.069656 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:14.069615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:14.379144 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:14.379065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:14.438696 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:14.438655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:17.006799 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:17.006757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:17.100650 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:17.100620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:17.249355 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:17.249318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:17.349442 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:17.349375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:17.680331 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:17.680236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:17.783785 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:17.783752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:21.775614 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:21.775581    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:06:42.222106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:42.222066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:42.611685 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:42.611653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:42.992642 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:42.992565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:43.073843 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:43.073814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:46.136745 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:46.136712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:46.240783 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:46.240752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:46.361371 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:46.361341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:46.623314 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:46.623286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:46.897555 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:46.897470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:46.989526 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:46.989463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:06:53.471259 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:53.471224    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:06:56.883895 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:06:56.883856    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:07:11.292489 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:11.292441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:07:13.482031 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.481992    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"]
Jul 28 15:07:13.482505 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.482339    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95eadd2f-2965-4392-82a2-bde46505d39b" containerName="revoke-keys"
Jul 28 15:07:13.482505 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.482352    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="95eadd2f-2965-4392-82a2-bde46505d39b" containerName="revoke-keys"
Jul 28 15:07:13.482505 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.482425    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="95eadd2f-2965-4392-82a2-bde46505d39b" containerName="revoke-keys"
Jul 28 15:07:13.485431 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.485407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.488127 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.488103    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"default-dockercfg-fvfrb\""
Jul 28 15:07:13.488251 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.488104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"auth-test-model-fd42ef-kserve-self-signed-certs\""
Jul 28 15:07:13.488251 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.488104    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"openshift-service-ca.crt\""
Jul 28 15:07:13.488251 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.488110    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"kube-root-ca.crt\""
Jul 28 15:07:13.493844 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.493824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"]
Jul 28 15:07:13.505784 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.505761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:07:13.555669 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.555639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-home\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.555795 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.555707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-model-cache\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.555795 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.555762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-tmp-dir\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.555900 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.555811    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9e755-5670-4b93-87c9-783a89fb15d1-tls-certs\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.555900 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.555839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-dshm\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.555979 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.555902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkh4v\" (UniqueName: \"kubernetes.io/projected/6ff9e755-5670-4b93-87c9-783a89fb15d1-kube-api-access-zkh4v\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656383 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9e755-5670-4b93-87c9-783a89fb15d1-tls-certs\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656542 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-dshm\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656542 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkh4v\" (UniqueName: \"kubernetes.io/projected/6ff9e755-5670-4b93-87c9-783a89fb15d1-kube-api-access-zkh4v\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656542 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-home\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656542 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-model-cache\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656542 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-tmp-dir\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656890 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-tmp-dir\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656963 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-model-cache\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.656963 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.656952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-home\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.658654 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.658634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-dshm\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.658822 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.658803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9e755-5670-4b93-87c9-783a89fb15d1-tls-certs\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.663819 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.663799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkh4v\" (UniqueName: \"kubernetes.io/projected/6ff9e755-5670-4b93-87c9-783a89fb15d1-kube-api-access-zkh4v\") pod \"auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") " pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.795441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.795344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:13.926731 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:13.926684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"]
Jul 28 15:07:13.930106 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:07:13.930072    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ff9e755_5670_4b93_87c9_783a89fb15d1.slice/crio-18311bfeba9f330e35700de971967b2b3263717cdb66d6ae156d4f43de54319d WatchSource:0}: Error finding container 18311bfeba9f330e35700de971967b2b3263717cdb66d6ae156d4f43de54319d: Status 404 returned error can't find the container with id 18311bfeba9f330e35700de971967b2b3263717cdb66d6ae156d4f43de54319d
Jul 28 15:07:14.294802 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:14.294770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5" event={"ID":"6ff9e755-5670-4b93-87c9-783a89fb15d1","Type":"ContainerStarted","Data":"29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e"}
Jul 28 15:07:14.294802 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:14.294807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5" event={"ID":"6ff9e755-5670-4b93-87c9-783a89fb15d1","Type":"ContainerStarted","Data":"18311bfeba9f330e35700de971967b2b3263717cdb66d6ae156d4f43de54319d"}
Jul 28 15:07:14.312020 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:14.311962    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5" podStartSLOduration=1.31194973 podStartE2EDuration="1.31194973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:07:14.309890803 +0000 UTC m=+2677.756740111" watchObservedRunningTime="2026-07-28 15:07:14.31194973 +0000 UTC m=+2677.758799039"
Jul 28 15:07:16.780325 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:16.780287    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:07:23.796105 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:23.796060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:23.796526 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:23.796127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:23.808115 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:23.808085    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:24.350112 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:24.350087    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:07:37.775684 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.775649    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"]
Jul 28 15:07:37.779695 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.779676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:37.781871 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.781848    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"openshift-service-ca.crt\""
Jul 28 15:07:37.781951 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.781853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"kube-root-ca.crt\""
Jul 28 15:07:37.782510 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.782493    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"default-dockercfg-rqmjw\""
Jul 28 15:07:37.782591 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.782575    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"auth-test-model-bc713f-kserve-self-signed-certs\""
Jul 28 15:07:37.789288 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.789268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"]
Jul 28 15:07:37.810030 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.810005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:07:37.938049 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.937988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-model-cache\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:37.938194 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.938132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzt7f\" (UniqueName: \"kubernetes.io/projected/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-kube-api-access-dzt7f\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:37.938268 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.938194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-dshm\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:37.938268 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.938244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tls-certs\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:37.938404 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.938271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tmp-dir\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:37.938470 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:37.938420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-home\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039044 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.038985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-dshm\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039044 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.039015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tls-certs\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039044 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.039034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tmp-dir\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039259 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.039065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-home\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039259 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.039170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-model-cache\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039367 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.039267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzt7f\" (UniqueName: \"kubernetes.io/projected/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-kube-api-access-dzt7f\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039471 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.039451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-home\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039574 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.039557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tmp-dir\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.039739 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.039721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-model-cache\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.041673 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.041653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-dshm\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.041942 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.041924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tls-certs\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.046586 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.046567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzt7f\" (UniqueName: \"kubernetes.io/projected/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-kube-api-access-dzt7f\") pod \"auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") " pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.090177 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.090160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:38.219430 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.219405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"]
Jul 28 15:07:38.221943 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:07:38.221912    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e3e3f67_ecf9_4f62_abc8_07a089653fbc.slice/crio-a509f9af14e730b36d7a1ebbb3f18a6fd6291b99b3640f52e5542139fa2de14e WatchSource:0}: Error finding container a509f9af14e730b36d7a1ebbb3f18a6fd6291b99b3640f52e5542139fa2de14e: Status 404 returned error can't find the container with id a509f9af14e730b36d7a1ebbb3f18a6fd6291b99b3640f52e5542139fa2de14e
Jul 28 15:07:38.405124 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.405089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx" event={"ID":"5e3e3f67-ecf9-4f62-abc8-07a089653fbc","Type":"ContainerStarted","Data":"344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6"}
Jul 28 15:07:38.405124 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.405129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx" event={"ID":"5e3e3f67-ecf9-4f62-abc8-07a089653fbc","Type":"ContainerStarted","Data":"a509f9af14e730b36d7a1ebbb3f18a6fd6291b99b3640f52e5542139fa2de14e"}
Jul 28 15:07:38.421548 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:38.421491    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx" podStartSLOduration=1.4214737419999999 podStartE2EDuration="1.421473742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:07:38.419187035 +0000 UTC m=+2701.866036343" watchObservedRunningTime="2026-07-28 15:07:38.421473742 +0000 UTC m=+2701.868323050"
Jul 28 15:07:41.669761 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:41.669721    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:07:48.090562 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:48.090526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:48.090983 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:48.090575    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:48.102343 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:48.102317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:48.466881 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:48.466794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:07:50.882036 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:07:50.881994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:08:07.281216 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:08:07.281186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:08:28.988877 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:08:28.988841    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:08:46.879883 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:08:46.879849    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:08:48.618204 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:08:48.618162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:08:48.718684 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:08:48.718652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:09:06.320968 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:09:06.320881    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:09:06.437443 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:09:06.437409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:09:09.772629 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:09:09.772588    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:09:27.978099 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:09:27.978058    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:09:33.435843 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:09:33.435805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:10:00.078447 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:10:00.078409    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:10:16.470885 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:10:16.470853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:10:36.670764 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:10:36.670735    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:10:37.326010 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:10:37.325979    2579 scope.go:117] "RemoveContainer" containerID="747c3a78f83dcbcf6afa774bc6a6ce1d2ea553859c8d8c1d9571ba4f8098e86a"
Jul 28 15:10:53.178546 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:10:53.178509    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:11:16.878662 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:11:16.878614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:11:35.582529 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:11:35.582493    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:11:37.345131 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:11:37.345095    2579 scope.go:117] "RemoveContainer" containerID="3f3714180696ef40c2561eb7f0cd4a710b1d135ac013ed75074669b9f39a1aa1"
Jul 28 15:12:01.574186 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:01.574105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:12:05.291898 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:05.291860    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"]
Jul 28 15:12:05.327645 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:05.327600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:12:06.260573 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:06.260540    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"]
Jul 28 15:12:06.269867 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:06.269840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:12:18.080248 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:18.080208    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:12:20.310854 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.310817    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5" podUID="6ff9e755-5670-4b93-87c9-783a89fb15d1" containerName="main" containerID="cri-o://29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e" gracePeriod=45
Jul 28 15:12:20.558871 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.558846    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:12:20.669297 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkh4v\" (UniqueName: \"kubernetes.io/projected/6ff9e755-5670-4b93-87c9-783a89fb15d1-kube-api-access-zkh4v\") pod \"6ff9e755-5670-4b93-87c9-783a89fb15d1\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") "
Jul 28 15:12:20.669297 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669273    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-dshm\") pod \"6ff9e755-5670-4b93-87c9-783a89fb15d1\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") "
Jul 28 15:12:20.669485 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-tmp-dir\") pod \"6ff9e755-5670-4b93-87c9-783a89fb15d1\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") "
Jul 28 15:12:20.669485 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669353    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-home\") pod \"6ff9e755-5670-4b93-87c9-783a89fb15d1\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") "
Jul 28 15:12:20.669485 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669423    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9e755-5670-4b93-87c9-783a89fb15d1-tls-certs\") pod \"6ff9e755-5670-4b93-87c9-783a89fb15d1\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") "
Jul 28 15:12:20.669485 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-model-cache\") pod \"6ff9e755-5670-4b93-87c9-783a89fb15d1\" (UID: \"6ff9e755-5670-4b93-87c9-783a89fb15d1\") "
Jul 28 15:12:20.669669 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669610    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-home" (OuterVolumeSpecName: "home") pod "6ff9e755-5670-4b93-87c9-783a89fb15d1" (UID: "6ff9e755-5670-4b93-87c9-783a89fb15d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:12:20.669669 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669633    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ff9e755-5670-4b93-87c9-783a89fb15d1" (UID: "6ff9e755-5670-4b93-87c9-783a89fb15d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:12:20.669788 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669760    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:20.669788 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669783    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:20.669884 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.669807    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ff9e755-5670-4b93-87c9-783a89fb15d1" (UID: "6ff9e755-5670-4b93-87c9-783a89fb15d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:12:20.671573 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.671540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff9e755-5670-4b93-87c9-783a89fb15d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ff9e755-5670-4b93-87c9-783a89fb15d1" (UID: "6ff9e755-5670-4b93-87c9-783a89fb15d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:12:20.671687 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.671607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff9e755-5670-4b93-87c9-783a89fb15d1-kube-api-access-zkh4v" (OuterVolumeSpecName: "kube-api-access-zkh4v") pod "6ff9e755-5670-4b93-87c9-783a89fb15d1" (UID: "6ff9e755-5670-4b93-87c9-783a89fb15d1"). InnerVolumeSpecName "kube-api-access-zkh4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:12:20.671687 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.671621    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-dshm" (OuterVolumeSpecName: "dshm") pod "6ff9e755-5670-4b93-87c9-783a89fb15d1" (UID: "6ff9e755-5670-4b93-87c9-783a89fb15d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:12:20.770869 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.770848    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:20.770990 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.770873    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkh4v\" (UniqueName: \"kubernetes.io/projected/6ff9e755-5670-4b93-87c9-783a89fb15d1-kube-api-access-zkh4v\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:20.770990 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.770887    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ff9e755-5670-4b93-87c9-783a89fb15d1-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:20.770990 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.770896    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff9e755-5670-4b93-87c9-783a89fb15d1-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:20.771928 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.771905    2579 generic.go:356] "Generic (PLEG): container finished" podID="6ff9e755-5670-4b93-87c9-783a89fb15d1" containerID="29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e" exitCode=0
Jul 28 15:12:20.772041 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.771961    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"
Jul 28 15:12:20.772041 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.771999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5" event={"ID":"6ff9e755-5670-4b93-87c9-783a89fb15d1","Type":"ContainerDied","Data":"29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e"}
Jul 28 15:12:20.772151 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.772042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5" event={"ID":"6ff9e755-5670-4b93-87c9-783a89fb15d1","Type":"ContainerDied","Data":"18311bfeba9f330e35700de971967b2b3263717cdb66d6ae156d4f43de54319d"}
Jul 28 15:12:20.772151 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.772061    2579 scope.go:117] "RemoveContainer" containerID="29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e"
Jul 28 15:12:20.780305 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.780288    2579 scope.go:117] "RemoveContainer" containerID="29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e"
Jul 28 15:12:20.780632 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:12:20.780611    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e\": container with ID starting with 29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e not found: ID does not exist" containerID="29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e"
Jul 28 15:12:20.780696 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.780639    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e"} err="failed to get container status \"29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e\": rpc error: code = NotFound desc = could not find container \"29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e\": container with ID starting with 29de5311714dbdd32d57f4a0644c64ed2d46aa3b2b8dada8bfae29477c140d8e not found: ID does not exist"
Jul 28 15:12:20.791683 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.791659    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"]
Jul 28 15:12:20.793158 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:20.793139    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/auth-test-model-fd42ef-kserve-6c7b95b999-fb4v5"]
Jul 28 15:12:21.251566 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.251532    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff9e755-5670-4b93-87c9-783a89fb15d1" path="/var/lib/kubelet/pods/6ff9e755-5670-4b93-87c9-783a89fb15d1/volumes"
Jul 28 15:12:21.280018 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.279961    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx" podUID="5e3e3f67-ecf9-4f62-abc8-07a089653fbc" containerName="main" containerID="cri-o://344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6" gracePeriod=45
Jul 28 15:12:21.540093 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.540064    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:12:21.577018 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.576987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-model-cache\") pod \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") "
Jul 28 15:12:21.577190 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.577052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-dshm\") pod \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") "
Jul 28 15:12:21.577190 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.577076    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tls-certs\") pod \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") "
Jul 28 15:12:21.577190 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.577131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-home\") pod \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") "
Jul 28 15:12:21.577190 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.577164    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzt7f\" (UniqueName: \"kubernetes.io/projected/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-kube-api-access-dzt7f\") pod \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") "
Jul 28 15:12:21.577462 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.577203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tmp-dir\") pod \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\" (UID: \"5e3e3f67-ecf9-4f62-abc8-07a089653fbc\") "
Jul 28 15:12:21.577618 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.577569    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-home" (OuterVolumeSpecName: "home") pod "5e3e3f67-ecf9-4f62-abc8-07a089653fbc" (UID: "5e3e3f67-ecf9-4f62-abc8-07a089653fbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:12:21.577743 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.577681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e3e3f67-ecf9-4f62-abc8-07a089653fbc" (UID: "5e3e3f67-ecf9-4f62-abc8-07a089653fbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:12:21.577743 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.577725    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e3e3f67-ecf9-4f62-abc8-07a089653fbc" (UID: "5e3e3f67-ecf9-4f62-abc8-07a089653fbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:12:21.579809 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.579785    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"]
Jul 28 15:12:21.580251 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580230    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e3e3f67-ecf9-4f62-abc8-07a089653fbc" containerName="main"
Jul 28 15:12:21.580251 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580253    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e3e3f67-ecf9-4f62-abc8-07a089653fbc" containerName="main"
Jul 28 15:12:21.580441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580263    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e3e3f67-ecf9-4f62-abc8-07a089653fbc" (UID: "5e3e3f67-ecf9-4f62-abc8-07a089653fbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:12:21.580441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580285    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff9e755-5670-4b93-87c9-783a89fb15d1" containerName="main"
Jul 28 15:12:21.580441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580294    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff9e755-5670-4b93-87c9-783a89fb15d1" containerName="main"
Jul 28 15:12:21.580441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580356    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff9e755-5670-4b93-87c9-783a89fb15d1" containerName="main"
Jul 28 15:12:21.580441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580365    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e3e3f67-ecf9-4f62-abc8-07a089653fbc" containerName="main"
Jul 28 15:12:21.580441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-dshm" (OuterVolumeSpecName: "dshm") pod "5e3e3f67-ecf9-4f62-abc8-07a089653fbc" (UID: "5e3e3f67-ecf9-4f62-abc8-07a089653fbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:12:21.580441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.580435    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-kube-api-access-dzt7f" (OuterVolumeSpecName: "kube-api-access-dzt7f") pod "5e3e3f67-ecf9-4f62-abc8-07a089653fbc" (UID: "5e3e3f67-ecf9-4f62-abc8-07a089653fbc"). InnerVolumeSpecName "kube-api-access-dzt7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:12:21.583163 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.583147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.585035 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.585017    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"openshift-service-ca.crt\""
Jul 28 15:12:21.585139 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.585086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"default-dockercfg-fvfrb\""
Jul 28 15:12:21.585139 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.585086    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"kube-root-ca.crt\""
Jul 28 15:12:21.585247 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.585232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"sub-test-model-fd42ef-kserve-self-signed-certs\""
Jul 28 15:12:21.598167 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.596532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"]
Jul 28 15:12:21.603761 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.603729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:12:21.678295 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29539d43-7e6b-4af4-99e6-a9f11961cfda-tls-certs\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.678417 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-home\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.678417 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-dshm\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.678510 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-model-cache\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.678510 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-tmp-dir\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.678510 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vr9\" (UniqueName: \"kubernetes.io/projected/29539d43-7e6b-4af4-99e6-a9f11961cfda-kube-api-access-27vr9\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.678621 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678557    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:21.678621 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678568    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:21.678621 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678576    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:21.678621 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678584    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:21.678621 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678592    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzt7f\" (UniqueName: \"kubernetes.io/projected/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-kube-api-access-dzt7f\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:21.678621 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.678601    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3e3f67-ecf9-4f62-abc8-07a089653fbc-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:12:21.778133 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.778060    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e3e3f67-ecf9-4f62-abc8-07a089653fbc" containerID="344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6" exitCode=0
Jul 28 15:12:21.778133 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.778118    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"
Jul 28 15:12:21.778324 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.778137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx" event={"ID":"5e3e3f67-ecf9-4f62-abc8-07a089653fbc","Type":"ContainerDied","Data":"344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6"}
Jul 28 15:12:21.778324 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.778173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx" event={"ID":"5e3e3f67-ecf9-4f62-abc8-07a089653fbc","Type":"ContainerDied","Data":"a509f9af14e730b36d7a1ebbb3f18a6fd6291b99b3640f52e5542139fa2de14e"}
Jul 28 15:12:21.778324 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.778192    2579 scope.go:117] "RemoveContainer" containerID="344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6"
Jul 28 15:12:21.778939 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.778915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-model-cache\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.779054 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.778942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-tmp-dir\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.779054 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.778966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vr9\" (UniqueName: \"kubernetes.io/projected/29539d43-7e6b-4af4-99e6-a9f11961cfda-kube-api-access-27vr9\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.779054 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.779035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29539d43-7e6b-4af4-99e6-a9f11961cfda-tls-certs\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.779226 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.779059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-home\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.779226 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.779082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-dshm\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.779418 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.779316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-model-cache\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.779624 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.779599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-home\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.779740 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.779654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-tmp-dir\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.781570 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.781540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-dshm\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.781687 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.781673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29539d43-7e6b-4af4-99e6-a9f11961cfda-tls-certs\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.785694 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.785675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vr9\" (UniqueName: \"kubernetes.io/projected/29539d43-7e6b-4af4-99e6-a9f11961cfda-kube-api-access-27vr9\") pod \"sub-test-model-fd42ef-kserve-767c966757-4mggs\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") " pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:21.786875 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.786857    2579 scope.go:117] "RemoveContainer" containerID="344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6"
Jul 28 15:12:21.787126 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:12:21.787109    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6\": container with ID starting with 344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6 not found: ID does not exist" containerID="344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6"
Jul 28 15:12:21.787199 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.787131    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6"} err="failed to get container status \"344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6\": rpc error: code = NotFound desc = could not find container \"344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6\": container with ID starting with 344f6f22c3cfd04b079f819badba50994af2b3c2eb1c3971c924b6c809b1f1c6 not found: ID does not exist"
Jul 28 15:12:21.797764 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.797741    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"]
Jul 28 15:12:21.804615 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.804595    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/auth-test-model-bc713f-kserve-6d9c497fd7-xb4cx"]
Jul 28 15:12:21.910586 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:21.910565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:22.039711 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:22.039675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"]
Jul 28 15:12:22.041989 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:12:22.041966    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29539d43_7e6b_4af4_99e6_a9f11961cfda.slice/crio-4ddf5ba6ce67c236c7cc189208de7e60402810bc6267441b875f6b6ae7020529 WatchSource:0}: Error finding container 4ddf5ba6ce67c236c7cc189208de7e60402810bc6267441b875f6b6ae7020529: Status 404 returned error can't find the container with id 4ddf5ba6ce67c236c7cc189208de7e60402810bc6267441b875f6b6ae7020529
Jul 28 15:12:22.785671 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:22.785635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs" event={"ID":"29539d43-7e6b-4af4-99e6-a9f11961cfda","Type":"ContainerStarted","Data":"c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658"}
Jul 28 15:12:22.786097 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:22.785680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs" event={"ID":"29539d43-7e6b-4af4-99e6-a9f11961cfda","Type":"ContainerStarted","Data":"4ddf5ba6ce67c236c7cc189208de7e60402810bc6267441b875f6b6ae7020529"}
Jul 28 15:12:22.800712 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:22.800669    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs" podStartSLOduration=1.8006532210000001 podStartE2EDuration="1.800653221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:12:22.799551857 +0000 UTC m=+2986.246401165" watchObservedRunningTime="2026-07-28 15:12:22.800653221 +0000 UTC m=+2986.247502529"
Jul 28 15:12:23.251735 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:23.251702    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e3e3f67-ecf9-4f62-abc8-07a089653fbc" path="/var/lib/kubelet/pods/5e3e3f67-ecf9-4f62-abc8-07a089653fbc/volumes"
Jul 28 15:12:25.198287 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:25.198256    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:12:28.102716 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:28.102684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:12:31.911153 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:31.911120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:31.911153 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:31.911158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:31.924054 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:31.924024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:32.844203 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:32.844174    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:12:39.593160 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.593130    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"]
Jul 28 15:12:39.597785 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.597762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.599989 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.599965    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"openshift-service-ca.crt\""
Jul 28 15:12:39.600776 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.600758    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"kube-root-ca.crt\""
Jul 28 15:12:39.600885 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.600773    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"sub-test-model-bc713f-kserve-self-signed-certs\""
Jul 28 15:12:39.600885 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.600818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"default-dockercfg-rqmjw\""
Jul 28 15:12:39.605727 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.605709    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"]
Jul 28 15:12:39.618031 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.618006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:12:39.694977 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.694953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-dshm\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.695123 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.694992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-model-cache\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.695123 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.695009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-home\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.695123 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.695029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztxj\" (UniqueName: \"kubernetes.io/projected/32fbbf25-ddc1-4911-bf9d-dc41c331951f-kube-api-access-zztxj\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.695244 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.695126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tls-certs\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.695244 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.695162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tmp-dir\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796341 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-dshm\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796490 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-model-cache\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796490 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-home\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796568 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztxj\" (UniqueName: \"kubernetes.io/projected/32fbbf25-ddc1-4911-bf9d-dc41c331951f-kube-api-access-zztxj\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796568 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tls-certs\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796647 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tmp-dir\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796762 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-home\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796811 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-model-cache\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.796927 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.796909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tmp-dir\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.798533 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.798514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-dshm\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.798844 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.798825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tls-certs\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.802835 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.802817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztxj\" (UniqueName: \"kubernetes.io/projected/32fbbf25-ddc1-4911-bf9d-dc41c331951f-kube-api-access-zztxj\") pod \"sub-test-model-bc713f-kserve-5868b479f5-zqkgl\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") " pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:39.908939 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:39.908859    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:40.037909 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:40.037792    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"]
Jul 28 15:12:40.040647 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:12:40.040619    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32fbbf25_ddc1_4911_bf9d_dc41c331951f.slice/crio-bf7cf2f30d197c674db3e3c135d51b4c7551289b1acd8da3fd44b4e600ad95f1 WatchSource:0}: Error finding container bf7cf2f30d197c674db3e3c135d51b4c7551289b1acd8da3fd44b4e600ad95f1: Status 404 returned error can't find the container with id bf7cf2f30d197c674db3e3c135d51b4c7551289b1acd8da3fd44b4e600ad95f1
Jul 28 15:12:40.869848 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:40.869812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl" event={"ID":"32fbbf25-ddc1-4911-bf9d-dc41c331951f","Type":"ContainerStarted","Data":"242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1"}
Jul 28 15:12:40.870348 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:40.870313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl" event={"ID":"32fbbf25-ddc1-4911-bf9d-dc41c331951f","Type":"ContainerStarted","Data":"bf7cf2f30d197c674db3e3c135d51b4c7551289b1acd8da3fd44b4e600ad95f1"}
Jul 28 15:12:40.887025 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:40.886985    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl" podStartSLOduration=1.8869710259999999 podStartE2EDuration="1.886971026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:12:40.885199207 +0000 UTC m=+3004.332048517" watchObservedRunningTime="2026-07-28 15:12:40.886971026 +0000 UTC m=+3004.333820335"
Jul 28 15:12:43.478070 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:43.478040    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:12:49.909308 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:49.909260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:49.909806 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:49.909346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:49.921413 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:49.921368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:12:50.932249 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:12:50.932216    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:13:07.217059 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:07.217024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:13:16.781508 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:16.781470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:13:30.173185 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:30.173099    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:13:43.830433 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:43.830383    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"]
Jul 28 15:13:43.838948 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:43.838924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:13:44.591413 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:44.591351    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"]
Jul 28 15:13:44.609492 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:44.609458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:13:47.871066 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:47.871027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:13:49.285708 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.285674    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"]
Jul 28 15:13:49.288382 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.288364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.290376 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.290354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fd42ef\"/\"rate-test-model-fd42ef-kserve-self-signed-certs\""
Jul 28 15:13:49.297666 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.297645    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"]
Jul 28 15:13:49.305853 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.305829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:13:49.409681 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.409654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-home\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.409804 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.409695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tls-certs\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.409804 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.409719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-dshm\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.409804 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.409748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2mw\" (UniqueName: \"kubernetes.io/projected/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-kube-api-access-gg2mw\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.409804 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.409776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tmp-dir\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.410006 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.409882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-model-cache\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510276 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tls-certs\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510413 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-dshm\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510413 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2mw\" (UniqueName: \"kubernetes.io/projected/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-kube-api-access-gg2mw\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510413 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tmp-dir\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510413 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-model-cache\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510611 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-home\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510797 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tmp-dir\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510892 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-home\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.510892 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.510841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-model-cache\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.512508 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.512487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-dshm\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.512755 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.512738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tls-certs\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.517840 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.517820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2mw\" (UniqueName: \"kubernetes.io/projected/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-kube-api-access-gg2mw\") pod \"rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") " pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.597644 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.597624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:49.724567 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:49.724533    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"]
Jul 28 15:13:49.725898 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:13:49.725862    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5e4dc84_57f9_4032_8eb0_7e24ae94842a.slice/crio-44f2500698a387143d979e9e5453ddbc6b216ec13d2f06db5e186629d871dd0b WatchSource:0}: Error finding container 44f2500698a387143d979e9e5453ddbc6b216ec13d2f06db5e186629d871dd0b: Status 404 returned error can't find the container with id 44f2500698a387143d979e9e5453ddbc6b216ec13d2f06db5e186629d871dd0b
Jul 28 15:13:50.195412 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:50.195365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg" event={"ID":"e5e4dc84-57f9-4032-8eb0-7e24ae94842a","Type":"ContainerStarted","Data":"378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f"}
Jul 28 15:13:50.195593 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:50.195426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg" event={"ID":"e5e4dc84-57f9-4032-8eb0-7e24ae94842a","Type":"ContainerStarted","Data":"44f2500698a387143d979e9e5453ddbc6b216ec13d2f06db5e186629d871dd0b"}
Jul 28 15:13:50.210577 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:50.210532    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg" podStartSLOduration=1.210514664 podStartE2EDuration="1.210514664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:13:50.210197077 +0000 UTC m=+3073.657046384" watchObservedRunningTime="2026-07-28 15:13:50.210514664 +0000 UTC m=+3073.657363969"
Jul 28 15:13:59.184608 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.184562    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs" podUID="29539d43-7e6b-4af4-99e6-a9f11961cfda" containerName="main" containerID="cri-o://c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658" gracePeriod=45
Jul 28 15:13:59.438132 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.438076    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:13:59.585605 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29539d43-7e6b-4af4-99e6-a9f11961cfda-tls-certs\") pod \"29539d43-7e6b-4af4-99e6-a9f11961cfda\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") "
Jul 28 15:13:59.585756 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585610    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-home\") pod \"29539d43-7e6b-4af4-99e6-a9f11961cfda\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") "
Jul 28 15:13:59.585756 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-tmp-dir\") pod \"29539d43-7e6b-4af4-99e6-a9f11961cfda\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") "
Jul 28 15:13:59.585756 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585709    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-dshm\") pod \"29539d43-7e6b-4af4-99e6-a9f11961cfda\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") "
Jul 28 15:13:59.585892 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-model-cache\") pod \"29539d43-7e6b-4af4-99e6-a9f11961cfda\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") "
Jul 28 15:13:59.585892 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-home" (OuterVolumeSpecName: "home") pod "29539d43-7e6b-4af4-99e6-a9f11961cfda" (UID: "29539d43-7e6b-4af4-99e6-a9f11961cfda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:13:59.585892 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27vr9\" (UniqueName: \"kubernetes.io/projected/29539d43-7e6b-4af4-99e6-a9f11961cfda-kube-api-access-27vr9\") pod \"29539d43-7e6b-4af4-99e6-a9f11961cfda\" (UID: \"29539d43-7e6b-4af4-99e6-a9f11961cfda\") "
Jul 28 15:13:59.586057 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29539d43-7e6b-4af4-99e6-a9f11961cfda" (UID: "29539d43-7e6b-4af4-99e6-a9f11961cfda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:13:59.586057 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.585961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-model-cache" (OuterVolumeSpecName: "model-cache") pod "29539d43-7e6b-4af4-99e6-a9f11961cfda" (UID: "29539d43-7e6b-4af4-99e6-a9f11961cfda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:13:59.586165 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.586067    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.586165 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.586079    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.586165 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.586091    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.587742 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.587714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-dshm" (OuterVolumeSpecName: "dshm") pod "29539d43-7e6b-4af4-99e6-a9f11961cfda" (UID: "29539d43-7e6b-4af4-99e6-a9f11961cfda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:13:59.587742 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.587726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29539d43-7e6b-4af4-99e6-a9f11961cfda-kube-api-access-27vr9" (OuterVolumeSpecName: "kube-api-access-27vr9") pod "29539d43-7e6b-4af4-99e6-a9f11961cfda" (UID: "29539d43-7e6b-4af4-99e6-a9f11961cfda"). InnerVolumeSpecName "kube-api-access-27vr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:13:59.587882 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.587753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29539d43-7e6b-4af4-99e6-a9f11961cfda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29539d43-7e6b-4af4-99e6-a9f11961cfda" (UID: "29539d43-7e6b-4af4-99e6-a9f11961cfda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:13:59.597889 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.597871    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:59.597983 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.597968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:59.609516 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.609490    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl" podUID="32fbbf25-ddc1-4911-bf9d-dc41c331951f" containerName="main" containerID="cri-o://242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1" gracePeriod=45
Jul 28 15:13:59.609718 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.609687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:13:59.686724 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.686701    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vr9\" (UniqueName: \"kubernetes.io/projected/29539d43-7e6b-4af4-99e6-a9f11961cfda-kube-api-access-27vr9\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.686724 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.686722    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29539d43-7e6b-4af4-99e6-a9f11961cfda-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.686830 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.686734    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29539d43-7e6b-4af4-99e6-a9f11961cfda-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.846121 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.846101    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:13:59.988869 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.988784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tls-certs\") pod \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") "
Jul 28 15:13:59.988869 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.988832    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-model-cache\") pod \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") "
Jul 28 15:13:59.989108 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.988901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-home\") pod \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") "
Jul 28 15:13:59.989108 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.988927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tmp-dir\") pod \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") "
Jul 28 15:13:59.989108 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.988970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-dshm\") pod \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") "
Jul 28 15:13:59.989108 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.989002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zztxj\" (UniqueName: \"kubernetes.io/projected/32fbbf25-ddc1-4911-bf9d-dc41c331951f-kube-api-access-zztxj\") pod \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\" (UID: \"32fbbf25-ddc1-4911-bf9d-dc41c331951f\") "
Jul 28 15:13:59.989302 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.989121    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-home" (OuterVolumeSpecName: "home") pod "32fbbf25-ddc1-4911-bf9d-dc41c331951f" (UID: "32fbbf25-ddc1-4911-bf9d-dc41c331951f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:13:59.989302 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.989152    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-model-cache" (OuterVolumeSpecName: "model-cache") pod "32fbbf25-ddc1-4911-bf9d-dc41c331951f" (UID: "32fbbf25-ddc1-4911-bf9d-dc41c331951f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:13:59.989302 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.989228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32fbbf25-ddc1-4911-bf9d-dc41c331951f" (UID: "32fbbf25-ddc1-4911-bf9d-dc41c331951f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:13:59.989448 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.989313    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.989448 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.989328    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.989448 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.989338    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:13:59.990918 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.990896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-dshm" (OuterVolumeSpecName: "dshm") pod "32fbbf25-ddc1-4911-bf9d-dc41c331951f" (UID: "32fbbf25-ddc1-4911-bf9d-dc41c331951f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:13:59.991033 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.990954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fbbf25-ddc1-4911-bf9d-dc41c331951f-kube-api-access-zztxj" (OuterVolumeSpecName: "kube-api-access-zztxj") pod "32fbbf25-ddc1-4911-bf9d-dc41c331951f" (UID: "32fbbf25-ddc1-4911-bf9d-dc41c331951f"). InnerVolumeSpecName "kube-api-access-zztxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:13:59.991033 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:13:59.990978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32fbbf25-ddc1-4911-bf9d-dc41c331951f" (UID: "32fbbf25-ddc1-4911-bf9d-dc41c331951f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:14:00.090698 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.090659    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32fbbf25-ddc1-4911-bf9d-dc41c331951f-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:14:00.090698 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.090692    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zztxj\" (UniqueName: \"kubernetes.io/projected/32fbbf25-ddc1-4911-bf9d-dc41c331951f-kube-api-access-zztxj\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:14:00.090698 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.090704    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32fbbf25-ddc1-4911-bf9d-dc41c331951f-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:14:00.243445 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.243360    2579 generic.go:356] "Generic (PLEG): container finished" podID="32fbbf25-ddc1-4911-bf9d-dc41c331951f" containerID="242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1" exitCode=0
Jul 28 15:14:00.243445 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.243425    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"
Jul 28 15:14:00.243896 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.243421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl" event={"ID":"32fbbf25-ddc1-4911-bf9d-dc41c331951f","Type":"ContainerDied","Data":"242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1"}
Jul 28 15:14:00.243896 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.243561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl" event={"ID":"32fbbf25-ddc1-4911-bf9d-dc41c331951f","Type":"ContainerDied","Data":"bf7cf2f30d197c674db3e3c135d51b4c7551289b1acd8da3fd44b4e600ad95f1"}
Jul 28 15:14:00.243896 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.243588    2579 scope.go:117] "RemoveContainer" containerID="242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1"
Jul 28 15:14:00.244999 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.244807    2579 generic.go:356] "Generic (PLEG): container finished" podID="29539d43-7e6b-4af4-99e6-a9f11961cfda" containerID="c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658" exitCode=0
Jul 28 15:14:00.244999 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.244833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs" event={"ID":"29539d43-7e6b-4af4-99e6-a9f11961cfda","Type":"ContainerDied","Data":"c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658"}
Jul 28 15:14:00.244999 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.244854    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"
Jul 28 15:14:00.244999 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.244864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs" event={"ID":"29539d43-7e6b-4af4-99e6-a9f11961cfda","Type":"ContainerDied","Data":"4ddf5ba6ce67c236c7cc189208de7e60402810bc6267441b875f6b6ae7020529"}
Jul 28 15:14:00.253086 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.253066    2579 scope.go:117] "RemoveContainer" containerID="242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1"
Jul 28 15:14:00.253366 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:14:00.253344    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1\": container with ID starting with 242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1 not found: ID does not exist" containerID="242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1"
Jul 28 15:14:00.253496 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.253374    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1"} err="failed to get container status \"242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1\": rpc error: code = NotFound desc = could not find container \"242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1\": container with ID starting with 242d5fd8af095eb283cad3f99ecbe2314cb876222f6098662788a1a8c27526b1 not found: ID does not exist"
Jul 28 15:14:00.253496 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.253412    2579 scope.go:117] "RemoveContainer" containerID="c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658"
Jul 28 15:14:00.259241 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.259223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:14:00.261156 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.261140    2579 scope.go:117] "RemoveContainer" containerID="c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658"
Jul 28 15:14:00.261387 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:14:00.261369    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658\": container with ID starting with c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658 not found: ID does not exist" containerID="c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658"
Jul 28 15:14:00.261467 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.261408    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658"} err="failed to get container status \"c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658\": rpc error: code = NotFound desc = could not find container \"c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658\": container with ID starting with c1488745c11add317b43e81dfb658a42da7213a7f4900328b7fa96ae9bc16658 not found: ID does not exist"
Jul 28 15:14:00.264256 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.264237    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"]
Jul 28 15:14:00.267838 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.267817    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/sub-test-model-bc713f-kserve-5868b479f5-zqkgl"]
Jul 28 15:14:00.281799 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.281776    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"]
Jul 28 15:14:00.284749 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:00.284724    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/sub-test-model-fd42ef-kserve-767c966757-4mggs"]
Jul 28 15:14:01.251863 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:01.251833    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29539d43-7e6b-4af4-99e6-a9f11961cfda" path="/var/lib/kubelet/pods/29539d43-7e6b-4af4-99e6-a9f11961cfda/volumes"
Jul 28 15:14:01.252227 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:01.252189    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32fbbf25-ddc1-4911-bf9d-dc41c331951f" path="/var/lib/kubelet/pods/32fbbf25-ddc1-4911-bf9d-dc41c331951f/volumes"
Jul 28 15:14:01.681295 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:01.681257    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:14:59.482964 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.482887    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"]
Jul 28 15:14:59.483439 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.483425    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29539d43-7e6b-4af4-99e6-a9f11961cfda" containerName="main"
Jul 28 15:14:59.483439 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.483440    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29539d43-7e6b-4af4-99e6-a9f11961cfda" containerName="main"
Jul 28 15:14:59.483523 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.483453    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32fbbf25-ddc1-4911-bf9d-dc41c331951f" containerName="main"
Jul 28 15:14:59.483523 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.483459    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32fbbf25-ddc1-4911-bf9d-dc41c331951f" containerName="main"
Jul 28 15:14:59.483589 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.483537    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29539d43-7e6b-4af4-99e6-a9f11961cfda" containerName="main"
Jul 28 15:14:59.483589 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.483548    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32fbbf25-ddc1-4911-bf9d-dc41c331951f" containerName="main"
Jul 28 15:14:59.487035 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.487014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.489928 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.489901    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"kube-root-ca.crt\""
Jul 28 15:14:59.490060 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.490028    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"rate-test-model-bc713f-kserve-self-signed-certs\""
Jul 28 15:14:59.490147 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.490055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"default-dockercfg-rqmjw\""
Jul 28 15:14:59.490206 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.490184    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc713f\"/\"openshift-service-ca.crt\""
Jul 28 15:14:59.498580 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.498557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"]
Jul 28 15:14:59.513425 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.513376    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:14:59.567064 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.567028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-dshm\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.567249 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.567070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-home\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.567249 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.567099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-tmp-dir\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.567249 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.567208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95vg\" (UniqueName: \"kubernetes.io/projected/df727959-9df7-4879-870b-398ba98d7d7e-kube-api-access-b95vg\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.567467 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.567274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df727959-9df7-4879-870b-398ba98d7d7e-tls-certs\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.567467 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.567328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-model-cache\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.667871 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.667840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-home\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.668063 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.667879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-tmp-dir\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.668063 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.668003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95vg\" (UniqueName: \"kubernetes.io/projected/df727959-9df7-4879-870b-398ba98d7d7e-kube-api-access-b95vg\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.668165 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.668080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df727959-9df7-4879-870b-398ba98d7d7e-tls-certs\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.668165 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.668117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-model-cache\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.668268 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.668197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-dshm\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.668268 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.668239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-tmp-dir\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.668425 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.668283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-home\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.668566 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.668543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-model-cache\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.670489 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.670467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-dshm\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.670638 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.670619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df727959-9df7-4879-870b-398ba98d7d7e-tls-certs\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.675143 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.675122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95vg\" (UniqueName: \"kubernetes.io/projected/df727959-9df7-4879-870b-398ba98d7d7e-kube-api-access-b95vg\") pod \"rate-test-model-bc713f-kserve-5467588547-nbtdj\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") " pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.797331 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.797297    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:14:59.929997 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:14:59.929900    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"]
Jul 28 15:14:59.932664 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:14:59.932633    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf727959_9df7_4879_870b_398ba98d7d7e.slice/crio-c7bf0f6dabcd3a3f01ea45513cec699d965fd187b5a4190c70d30df58d2e653a WatchSource:0}: Error finding container c7bf0f6dabcd3a3f01ea45513cec699d965fd187b5a4190c70d30df58d2e653a: Status 404 returned error can't find the container with id c7bf0f6dabcd3a3f01ea45513cec699d965fd187b5a4190c70d30df58d2e653a
Jul 28 15:15:00.131315 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.131216    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"]
Jul 28 15:15:00.135028 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.135007    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"]
Jul 28 15:15:00.135144 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.135127    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"
Jul 28 15:15:00.137111 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.137092    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nx5zp\""
Jul 28 15:15:00.138197 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.138182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"
Jul 28 15:15:00.144353 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.144330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"]
Jul 28 15:15:00.146268 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.146245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"]
Jul 28 15:15:00.148698 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.148681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"
Jul 28 15:15:00.156030 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.156007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"
Jul 28 15:15:00.288936 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.288902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"]
Jul 28 15:15:00.293918 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:15:00.293886    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfb0510_1b88_49c7_b9d8_3568dc21fcf6.slice/crio-47ff125bfe4c59309ac219a0e5075653193c0002fb686681e0b0112e2a09ad08 WatchSource:0}: Error finding container 47ff125bfe4c59309ac219a0e5075653193c0002fb686681e0b0112e2a09ad08: Status 404 returned error can't find the container with id 47ff125bfe4c59309ac219a0e5075653193c0002fb686681e0b0112e2a09ad08
Jul 28 15:15:00.313687 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.313659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"]
Jul 28 15:15:00.324959 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:15:00.324932    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cd9a00_8f67_4653_a86a_b6392e8f644c.slice/crio-7e2696d2377fc15bb92b1b811f7080c681aadf1e9b445648f322aadbf8f75bc2 WatchSource:0}: Error finding container 7e2696d2377fc15bb92b1b811f7080c681aadf1e9b445648f322aadbf8f75bc2: Status 404 returned error can't find the container with id 7e2696d2377fc15bb92b1b811f7080c681aadf1e9b445648f322aadbf8f75bc2
Jul 28 15:15:00.523269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.523184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst" event={"ID":"0dfb0510-1b88-49c7-b9d8-3568dc21fcf6","Type":"ContainerStarted","Data":"47ff125bfe4c59309ac219a0e5075653193c0002fb686681e0b0112e2a09ad08"}
Jul 28 15:15:00.525027 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.524942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8" event={"ID":"d3cd9a00-8f67-4653-a86a-b6392e8f644c","Type":"ContainerStarted","Data":"7e2696d2377fc15bb92b1b811f7080c681aadf1e9b445648f322aadbf8f75bc2"}
Jul 28 15:15:00.527139 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.527089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj" event={"ID":"df727959-9df7-4879-870b-398ba98d7d7e","Type":"ContainerStarted","Data":"06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55"}
Jul 28 15:15:00.527139 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.527129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj" event={"ID":"df727959-9df7-4879-870b-398ba98d7d7e","Type":"ContainerStarted","Data":"c7bf0f6dabcd3a3f01ea45513cec699d965fd187b5a4190c70d30df58d2e653a"}
Jul 28 15:15:00.544056 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:00.544013    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj" podStartSLOduration=1.543998185 podStartE2EDuration="1.543998185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:15:00.542117186 +0000 UTC m=+3143.988966494" watchObservedRunningTime="2026-07-28 15:15:00.543998185 +0000 UTC m=+3143.990847501"
Jul 28 15:15:01.533488 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:01.533452    2579 generic.go:356] "Generic (PLEG): container finished" podID="0dfb0510-1b88-49c7-b9d8-3568dc21fcf6" containerID="f2be4c30a15988b95eaf6b188b187da9e289f08e4eaa1e146c72d27337dcfb47" exitCode=0
Jul 28 15:15:01.533866 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:01.533540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst" event={"ID":"0dfb0510-1b88-49c7-b9d8-3568dc21fcf6","Type":"ContainerDied","Data":"f2be4c30a15988b95eaf6b188b187da9e289f08e4eaa1e146c72d27337dcfb47"}
Jul 28 15:15:01.535037 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:01.535013    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3cd9a00-8f67-4653-a86a-b6392e8f644c" containerID="54da89c32dcb00e623a1e0c19539a93fb37439e228c76be519ef37e4d41acf49" exitCode=0
Jul 28 15:15:01.535156 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:01.535100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8" event={"ID":"d3cd9a00-8f67-4653-a86a-b6392e8f644c","Type":"ContainerDied","Data":"54da89c32dcb00e623a1e0c19539a93fb37439e228c76be519ef37e4d41acf49"}
Jul 28 15:15:02.686671 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:02.686644    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"
Jul 28 15:15:02.689639 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:02.689621    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"
Jul 28 15:15:03.545648 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:03.545626    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"
Jul 28 15:15:03.545797 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:03.545629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst" event={"ID":"0dfb0510-1b88-49c7-b9d8-3568dc21fcf6","Type":"ContainerDied","Data":"47ff125bfe4c59309ac219a0e5075653193c0002fb686681e0b0112e2a09ad08"}
Jul 28 15:15:03.545797 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:03.545758    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47ff125bfe4c59309ac219a0e5075653193c0002fb686681e0b0112e2a09ad08"
Jul 28 15:15:03.546872 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:03.546851    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"
Jul 28 15:15:03.546993 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:03.546897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8" event={"ID":"d3cd9a00-8f67-4653-a86a-b6392e8f644c","Type":"ContainerDied","Data":"7e2696d2377fc15bb92b1b811f7080c681aadf1e9b445648f322aadbf8f75bc2"}
Jul 28 15:15:03.546993 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:03.546936    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2696d2377fc15bb92b1b811f7080c681aadf1e9b445648f322aadbf8f75bc2"
Jul 28 15:15:03.576449 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:03.576423    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:15:09.797837 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:09.797797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:15:09.798236 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:09.797888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:15:09.810090 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:09.810064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:15:10.590648 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:10.590620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:15:34.273111 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:34.273075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:15:40.980430 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:40.980359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:15:51.984189 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:15:51.984144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:16:12.280145 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:12.280110    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:16:19.493976 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:19.493934    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"]
Jul 28 15:16:19.516020 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:19.515988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:20.472160 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:20.472126    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"]
Jul 28 15:16:20.537002 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:20.536966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:21.465024 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.464983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7611-140335585398064"]
Jul 28 15:16:21.465428 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.465409    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3cd9a00-8f67-4653-a86a-b6392e8f644c" containerName="cleanup"
Jul 28 15:16:21.465535 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.465432    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3cd9a00-8f67-4653-a86a-b6392e8f644c" containerName="cleanup"
Jul 28 15:16:21.465535 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.465463    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dfb0510-1b88-49c7-b9d8-3568dc21fcf6" containerName="cleanup"
Jul 28 15:16:21.465535 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.465472    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dfb0510-1b88-49c7-b9d8-3568dc21fcf6" containerName="cleanup"
Jul 28 15:16:21.465694 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.465573    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3cd9a00-8f67-4653-a86a-b6392e8f644c" containerName="cleanup"
Jul 28 15:16:21.465694 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.465590    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dfb0510-1b88-49c7-b9d8-3568dc21fcf6" containerName="cleanup"
Jul 28 15:16:21.468986 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.468965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585398064"
Jul 28 15:16:21.470920 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.470901    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j86tw\""
Jul 28 15:16:21.474134 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.474109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585398064"]
Jul 28 15:16:21.545441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.545372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5d6\" (UniqueName: \"kubernetes.io/projected/230b3ba9-dafa-4b55-901c-225e45d99eed-kube-api-access-gb5d6\") pod \"test-curl-7611-140335585398064\" (UID: \"230b3ba9-dafa-4b55-901c-225e45d99eed\") " pod="opendatahub/test-curl-7611-140335585398064"
Jul 28 15:16:21.646193 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.646152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5d6\" (UniqueName: \"kubernetes.io/projected/230b3ba9-dafa-4b55-901c-225e45d99eed-kube-api-access-gb5d6\") pod \"test-curl-7611-140335585398064\" (UID: \"230b3ba9-dafa-4b55-901c-225e45d99eed\") " pod="opendatahub/test-curl-7611-140335585398064"
Jul 28 15:16:21.653670 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.653634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5d6\" (UniqueName: \"kubernetes.io/projected/230b3ba9-dafa-4b55-901c-225e45d99eed-kube-api-access-gb5d6\") pod \"test-curl-7611-140335585398064\" (UID: \"230b3ba9-dafa-4b55-901c-225e45d99eed\") " pod="opendatahub/test-curl-7611-140335585398064"
Jul 28 15:16:21.779846 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.779725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585398064"
Jul 28 15:16:21.924104 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.924078    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585398064"]
Jul 28 15:16:21.926980 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:16:21.926950    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod230b3ba9_dafa_4b55_901c_225e45d99eed.slice/crio-fa5a2948f6bfc2cac435fd19b3b38f836fc162d53fe853a42bd7d81b9648b2da WatchSource:0}: Error finding container fa5a2948f6bfc2cac435fd19b3b38f836fc162d53fe853a42bd7d81b9648b2da: Status 404 returned error can't find the container with id fa5a2948f6bfc2cac435fd19b3b38f836fc162d53fe853a42bd7d81b9648b2da
Jul 28 15:16:21.928636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:21.928616    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:16:22.935778 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:22.935671    2579 generic.go:356] "Generic (PLEG): container finished" podID="230b3ba9-dafa-4b55-901c-225e45d99eed" containerID="d4075dd4cd2c65f753ce26c610e301588047fcf00affb2aafe9784304929ae06" exitCode=0
Jul 28 15:16:22.935778 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:22.935757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585398064" event={"ID":"230b3ba9-dafa-4b55-901c-225e45d99eed","Type":"ContainerDied","Data":"d4075dd4cd2c65f753ce26c610e301588047fcf00affb2aafe9784304929ae06"}
Jul 28 15:16:22.936206 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:22.935812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585398064" event={"ID":"230b3ba9-dafa-4b55-901c-225e45d99eed","Type":"ContainerStarted","Data":"fa5a2948f6bfc2cac435fd19b3b38f836fc162d53fe853a42bd7d81b9648b2da"}
Jul 28 15:16:24.082832 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.082805    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585398064"
Jul 28 15:16:24.102048 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.102017    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7611-140335585398064_230b3ba9-dafa-4b55-901c-225e45d99eed/test-curl-7611-140335585398064/0.log"
Jul 28 15:16:24.129666 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.129617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7611-140335585398064"]
Jul 28 15:16:24.131681 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.131652    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7611-140335585398064"]
Jul 28 15:16:24.165929 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.165899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb5d6\" (UniqueName: \"kubernetes.io/projected/230b3ba9-dafa-4b55-901c-225e45d99eed-kube-api-access-gb5d6\") pod \"230b3ba9-dafa-4b55-901c-225e45d99eed\" (UID: \"230b3ba9-dafa-4b55-901c-225e45d99eed\") "
Jul 28 15:16:24.168269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.168241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230b3ba9-dafa-4b55-901c-225e45d99eed-kube-api-access-gb5d6" (OuterVolumeSpecName: "kube-api-access-gb5d6") pod "230b3ba9-dafa-4b55-901c-225e45d99eed" (UID: "230b3ba9-dafa-4b55-901c-225e45d99eed"). InnerVolumeSpecName "kube-api-access-gb5d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:16:24.250824 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.250737    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7611-140335585312816"]
Jul 28 15:16:24.251083 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.251071    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230b3ba9-dafa-4b55-901c-225e45d99eed" containerName="test-curl-7611-140335585398064"
Jul 28 15:16:24.251130 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.251084    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="230b3ba9-dafa-4b55-901c-225e45d99eed" containerName="test-curl-7611-140335585398064"
Jul 28 15:16:24.251166 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.251160    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="230b3ba9-dafa-4b55-901c-225e45d99eed" containerName="test-curl-7611-140335585398064"
Jul 28 15:16:24.254038 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.254020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585312816"
Jul 28 15:16:24.257929 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.257904    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585312816"]
Jul 28 15:16:24.266813 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.266777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hg8m\" (UniqueName: \"kubernetes.io/projected/4ea3ec24-052f-4505-85ef-2e7665094980-kube-api-access-5hg8m\") pod \"test-curl-7611-140335585312816\" (UID: \"4ea3ec24-052f-4505-85ef-2e7665094980\") " pod="opendatahub/test-curl-7611-140335585312816"
Jul 28 15:16:24.267053 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.267028    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5d6\" (UniqueName: \"kubernetes.io/projected/230b3ba9-dafa-4b55-901c-225e45d99eed-kube-api-access-gb5d6\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:24.368168 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.368127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hg8m\" (UniqueName: \"kubernetes.io/projected/4ea3ec24-052f-4505-85ef-2e7665094980-kube-api-access-5hg8m\") pod \"test-curl-7611-140335585312816\" (UID: \"4ea3ec24-052f-4505-85ef-2e7665094980\") " pod="opendatahub/test-curl-7611-140335585312816"
Jul 28 15:16:24.376149 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.376113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hg8m\" (UniqueName: \"kubernetes.io/projected/4ea3ec24-052f-4505-85ef-2e7665094980-kube-api-access-5hg8m\") pod \"test-curl-7611-140335585312816\" (UID: \"4ea3ec24-052f-4505-85ef-2e7665094980\") " pod="opendatahub/test-curl-7611-140335585312816"
Jul 28 15:16:24.564974 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.564933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585312816"
Jul 28 15:16:24.701154 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.701119    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585312816"]
Jul 28 15:16:24.704219 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:16:24.704182    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea3ec24_052f_4505_85ef_2e7665094980.slice/crio-2b15c003d5b16f50449500cfd1c0857124680ee6b79a6cadb89dde44fb157d20 WatchSource:0}: Error finding container 2b15c003d5b16f50449500cfd1c0857124680ee6b79a6cadb89dde44fb157d20: Status 404 returned error can't find the container with id 2b15c003d5b16f50449500cfd1c0857124680ee6b79a6cadb89dde44fb157d20
Jul 28 15:16:24.947910 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.947817    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585398064"
Jul 28 15:16:24.947910 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.947847    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa5a2948f6bfc2cac435fd19b3b38f836fc162d53fe853a42bd7d81b9648b2da"
Jul 28 15:16:24.949043 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:24.949007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585312816" event={"ID":"4ea3ec24-052f-4505-85ef-2e7665094980","Type":"ContainerStarted","Data":"2b15c003d5b16f50449500cfd1c0857124680ee6b79a6cadb89dde44fb157d20"}
Jul 28 15:16:24.958599 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:16:24.958559    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7611-140335585398064\" is forbidden: User \"system:node:ip-10-0-141-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-160.ec2.internal' and this object" podUID="230b3ba9-dafa-4b55-901c-225e45d99eed" pod="opendatahub/test-curl-7611-140335585398064"
Jul 28 15:16:25.252386 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:25.252298    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230b3ba9-dafa-4b55-901c-225e45d99eed" path="/var/lib/kubelet/pods/230b3ba9-dafa-4b55-901c-225e45d99eed/volumes"
Jul 28 15:16:25.783739 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:25.783700    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:16:25.954777 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:25.954741    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ea3ec24-052f-4505-85ef-2e7665094980" containerID="8174b2e6ac0d9f21d5e4975ad05c97bd39f9c61e05e68b788057e78ec1bc5bc9" exitCode=0
Jul 28 15:16:25.954959 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:25.954828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585312816" event={"ID":"4ea3ec24-052f-4505-85ef-2e7665094980","Type":"ContainerDied","Data":"8174b2e6ac0d9f21d5e4975ad05c97bd39f9c61e05e68b788057e78ec1bc5bc9"}
Jul 28 15:16:27.100192 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.100160    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585312816"
Jul 28 15:16:27.120032 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.119980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7611-140335585312816_4ea3ec24-052f-4505-85ef-2e7665094980/test-curl-7611-140335585312816/0.log"
Jul 28 15:16:27.150341 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.150299    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7611-140335585312816"]
Jul 28 15:16:27.150623 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.150605    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7611-140335585312816"]
Jul 28 15:16:27.190506 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.190466    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hg8m\" (UniqueName: \"kubernetes.io/projected/4ea3ec24-052f-4505-85ef-2e7665094980-kube-api-access-5hg8m\") pod \"4ea3ec24-052f-4505-85ef-2e7665094980\" (UID: \"4ea3ec24-052f-4505-85ef-2e7665094980\") "
Jul 28 15:16:27.192673 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.192646    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea3ec24-052f-4505-85ef-2e7665094980-kube-api-access-5hg8m" (OuterVolumeSpecName: "kube-api-access-5hg8m") pod "4ea3ec24-052f-4505-85ef-2e7665094980" (UID: "4ea3ec24-052f-4505-85ef-2e7665094980"). InnerVolumeSpecName "kube-api-access-5hg8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:16:27.252643 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.252567    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea3ec24-052f-4505-85ef-2e7665094980" path="/var/lib/kubelet/pods/4ea3ec24-052f-4505-85ef-2e7665094980/volumes"
Jul 28 15:16:27.291213 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.291175    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hg8m\" (UniqueName: \"kubernetes.io/projected/4ea3ec24-052f-4505-85ef-2e7665094980-kube-api-access-5hg8m\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:27.319982 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.319944    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:27.320325 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.320311    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea3ec24-052f-4505-85ef-2e7665094980" containerName="test-curl-7611-140335585312816"
Jul 28 15:16:27.320380 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.320327    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea3ec24-052f-4505-85ef-2e7665094980" containerName="test-curl-7611-140335585312816"
Jul 28 15:16:27.320436 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.320416    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea3ec24-052f-4505-85ef-2e7665094980" containerName="test-curl-7611-140335585312816"
Jul 28 15:16:27.323585 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.323566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:27.329360 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.329327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:27.392607 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.392554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2s78\" (UniqueName: \"kubernetes.io/projected/f7d368e1-40df-409a-b106-a3d71ecc21c3-kube-api-access-x2s78\") pod \"test-curl-7611-140335585417232\" (UID: \"f7d368e1-40df-409a-b106-a3d71ecc21c3\") " pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:27.493260 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.493213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2s78\" (UniqueName: \"kubernetes.io/projected/f7d368e1-40df-409a-b106-a3d71ecc21c3-kube-api-access-x2s78\") pod \"test-curl-7611-140335585417232\" (UID: \"f7d368e1-40df-409a-b106-a3d71ecc21c3\") " pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:27.501364 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.501333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2s78\" (UniqueName: \"kubernetes.io/projected/f7d368e1-40df-409a-b106-a3d71ecc21c3-kube-api-access-x2s78\") pod \"test-curl-7611-140335585417232\" (UID: \"f7d368e1-40df-409a-b106-a3d71ecc21c3\") " pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:27.633995 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.633955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:27.769870 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.769834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:27.772458 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:16:27.772427    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7d368e1_40df_409a_b106_a3d71ecc21c3.slice/crio-fb87d974b8d47aa7cee3fb11ce5d21847ae562a33db60d1740b33a81e51515a9 WatchSource:0}: Error finding container fb87d974b8d47aa7cee3fb11ce5d21847ae562a33db60d1740b33a81e51515a9: Status 404 returned error can't find the container with id fb87d974b8d47aa7cee3fb11ce5d21847ae562a33db60d1740b33a81e51515a9
Jul 28 15:16:27.966800 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.966706    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585312816"
Jul 28 15:16:27.966800 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.966713    2579 scope.go:117] "RemoveContainer" containerID="8174b2e6ac0d9f21d5e4975ad05c97bd39f9c61e05e68b788057e78ec1bc5bc9"
Jul 28 15:16:27.968196 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:27.967974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585417232" event={"ID":"f7d368e1-40df-409a-b106-a3d71ecc21c3","Type":"ContainerStarted","Data":"fb87d974b8d47aa7cee3fb11ce5d21847ae562a33db60d1740b33a81e51515a9"}
Jul 28 15:16:28.976541 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:28.976482    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7d368e1-40df-409a-b106-a3d71ecc21c3" containerID="c08bd29da9d19539dbbb307cbf6adec1b7f77287f16ca4f5e6fc337438b8fd12" exitCode=0
Jul 28 15:16:28.977121 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:28.976587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585417232" event={"ID":"f7d368e1-40df-409a-b106-a3d71ecc21c3","Type":"ContainerDied","Data":"c08bd29da9d19539dbbb307cbf6adec1b7f77287f16ca4f5e6fc337438b8fd12"}
Jul 28 15:16:30.121066 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.121034    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:30.138946 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.138918    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7611-140335585417232_f7d368e1-40df-409a-b106-a3d71ecc21c3/test-curl-7611-140335585417232/0.log"
Jul 28 15:16:30.163009 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.162975    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:30.168194 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.168170    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:30.216661 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.216631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2s78\" (UniqueName: \"kubernetes.io/projected/f7d368e1-40df-409a-b106-a3d71ecc21c3-kube-api-access-x2s78\") pod \"f7d368e1-40df-409a-b106-a3d71ecc21c3\" (UID: \"f7d368e1-40df-409a-b106-a3d71ecc21c3\") "
Jul 28 15:16:30.218786 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.218760    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d368e1-40df-409a-b106-a3d71ecc21c3-kube-api-access-x2s78" (OuterVolumeSpecName: "kube-api-access-x2s78") pod "f7d368e1-40df-409a-b106-a3d71ecc21c3" (UID: "f7d368e1-40df-409a-b106-a3d71ecc21c3"). InnerVolumeSpecName "kube-api-access-x2s78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:16:30.296045 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.296015    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:30.296363 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.296351    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d368e1-40df-409a-b106-a3d71ecc21c3" containerName="test-curl-7611-140335585417232"
Jul 28 15:16:30.296441 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.296365    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d368e1-40df-409a-b106-a3d71ecc21c3" containerName="test-curl-7611-140335585417232"
Jul 28 15:16:30.296478 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.296468    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d368e1-40df-409a-b106-a3d71ecc21c3" containerName="test-curl-7611-140335585417232"
Jul 28 15:16:30.299291 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.299274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:30.307713 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.307643    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:30.318160 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.318133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwnp\" (UniqueName: \"kubernetes.io/projected/5774fc3f-1975-4021-bef6-0d4a7fb9b2cb-kube-api-access-tpwnp\") pod \"test-curl-7611-140335585417232\" (UID: \"5774fc3f-1975-4021-bef6-0d4a7fb9b2cb\") " pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:30.318314 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.318189    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2s78\" (UniqueName: \"kubernetes.io/projected/f7d368e1-40df-409a-b106-a3d71ecc21c3-kube-api-access-x2s78\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:30.419120 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.419084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwnp\" (UniqueName: \"kubernetes.io/projected/5774fc3f-1975-4021-bef6-0d4a7fb9b2cb-kube-api-access-tpwnp\") pod \"test-curl-7611-140335585417232\" (UID: \"5774fc3f-1975-4021-bef6-0d4a7fb9b2cb\") " pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:30.426644 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.426608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwnp\" (UniqueName: \"kubernetes.io/projected/5774fc3f-1975-4021-bef6-0d4a7fb9b2cb-kube-api-access-tpwnp\") pod \"test-curl-7611-140335585417232\" (UID: \"5774fc3f-1975-4021-bef6-0d4a7fb9b2cb\") " pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:30.610920 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.610809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:30.748424 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.748357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:30.751078 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:16:30.751051    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5774fc3f_1975_4021_bef6_0d4a7fb9b2cb.slice/crio-0368b9e4959d2459f6ce4bdbf08ff1887365908510a39607e3a37f10bf2d79e6 WatchSource:0}: Error finding container 0368b9e4959d2459f6ce4bdbf08ff1887365908510a39607e3a37f10bf2d79e6: Status 404 returned error can't find the container with id 0368b9e4959d2459f6ce4bdbf08ff1887365908510a39607e3a37f10bf2d79e6
Jul 28 15:16:30.989948 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.989862    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:30.989948 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.989903    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb87d974b8d47aa7cee3fb11ce5d21847ae562a33db60d1740b33a81e51515a9"
Jul 28 15:16:30.991102 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:30.991075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585417232" event={"ID":"5774fc3f-1975-4021-bef6-0d4a7fb9b2cb","Type":"ContainerStarted","Data":"0368b9e4959d2459f6ce4bdbf08ff1887365908510a39607e3a37f10bf2d79e6"}
Jul 28 15:16:31.001082 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:31.001052    2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7611-140335585417232" oldPodUID="f7d368e1-40df-409a-b106-a3d71ecc21c3" podUID="5774fc3f-1975-4021-bef6-0d4a7fb9b2cb"
Jul 28 15:16:31.252825 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:31.252744    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d368e1-40df-409a-b106-a3d71ecc21c3" path="/var/lib/kubelet/pods/f7d368e1-40df-409a-b106-a3d71ecc21c3/volumes"
Jul 28 15:16:31.998068 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:31.998029    2579 generic.go:356] "Generic (PLEG): container finished" podID="5774fc3f-1975-4021-bef6-0d4a7fb9b2cb" containerID="116bfb75b253a9b4c95dbdf00836d29a1fa04a34850718072107e3f47b6bc4ca" exitCode=0
Jul 28 15:16:31.998230 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:31.998099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585417232" event={"ID":"5774fc3f-1975-4021-bef6-0d4a7fb9b2cb","Type":"ContainerDied","Data":"116bfb75b253a9b4c95dbdf00836d29a1fa04a34850718072107e3f47b6bc4ca"}
Jul 28 15:16:33.144238 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.144207    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:33.162426 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.162379    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7611-140335585417232_5774fc3f-1975-4021-bef6-0d4a7fb9b2cb/test-curl-7611-140335585417232/0.log"
Jul 28 15:16:33.185458 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.185415    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:33.189234 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.189208    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7611-140335585417232"]
Jul 28 15:16:33.245847 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.245811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpwnp\" (UniqueName: \"kubernetes.io/projected/5774fc3f-1975-4021-bef6-0d4a7fb9b2cb-kube-api-access-tpwnp\") pod \"5774fc3f-1975-4021-bef6-0d4a7fb9b2cb\" (UID: \"5774fc3f-1975-4021-bef6-0d4a7fb9b2cb\") "
Jul 28 15:16:33.247992 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.247959    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5774fc3f-1975-4021-bef6-0d4a7fb9b2cb-kube-api-access-tpwnp" (OuterVolumeSpecName: "kube-api-access-tpwnp") pod "5774fc3f-1975-4021-bef6-0d4a7fb9b2cb" (UID: "5774fc3f-1975-4021-bef6-0d4a7fb9b2cb"). InnerVolumeSpecName "kube-api-access-tpwnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:16:33.252032 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.251993    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5774fc3f-1975-4021-bef6-0d4a7fb9b2cb" path="/var/lib/kubelet/pods/5774fc3f-1975-4021-bef6-0d4a7fb9b2cb/volumes"
Jul 28 15:16:33.312843 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.312802    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:33.313169 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.313155    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5774fc3f-1975-4021-bef6-0d4a7fb9b2cb" containerName="test-curl-7611-140335585417232"
Jul 28 15:16:33.313232 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.313170    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5774fc3f-1975-4021-bef6-0d4a7fb9b2cb" containerName="test-curl-7611-140335585417232"
Jul 28 15:16:33.313269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.313232    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5774fc3f-1975-4021-bef6-0d4a7fb9b2cb" containerName="test-curl-7611-140335585417232"
Jul 28 15:16:33.316520 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.316500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:33.320222 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.320193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:33.347443 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.347383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8vp\" (UniqueName: \"kubernetes.io/projected/8f5fdfff-4031-4a87-8a7a-1d05eae2a753-kube-api-access-5l8vp\") pod \"test-curl-7611-140335585418240\" (UID: \"8f5fdfff-4031-4a87-8a7a-1d05eae2a753\") " pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:33.347706 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.347680    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpwnp\" (UniqueName: \"kubernetes.io/projected/5774fc3f-1975-4021-bef6-0d4a7fb9b2cb-kube-api-access-tpwnp\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:33.448692 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.448653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8vp\" (UniqueName: \"kubernetes.io/projected/8f5fdfff-4031-4a87-8a7a-1d05eae2a753-kube-api-access-5l8vp\") pod \"test-curl-7611-140335585418240\" (UID: \"8f5fdfff-4031-4a87-8a7a-1d05eae2a753\") " pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:33.456428 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.456377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8vp\" (UniqueName: \"kubernetes.io/projected/8f5fdfff-4031-4a87-8a7a-1d05eae2a753-kube-api-access-5l8vp\") pod \"test-curl-7611-140335585418240\" (UID: \"8f5fdfff-4031-4a87-8a7a-1d05eae2a753\") " pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:33.627028 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.626985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:33.764137 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:33.764101    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:33.766992 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:16:33.766965    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f5fdfff_4031_4a87_8a7a_1d05eae2a753.slice/crio-4e4dbfbeb6c72c3e4a3cda2a12d34614664b96a7e07a34114e53686df8aaf52d WatchSource:0}: Error finding container 4e4dbfbeb6c72c3e4a3cda2a12d34614664b96a7e07a34114e53686df8aaf52d: Status 404 returned error can't find the container with id 4e4dbfbeb6c72c3e4a3cda2a12d34614664b96a7e07a34114e53686df8aaf52d
Jul 28 15:16:34.010106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.010003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585418240" event={"ID":"8f5fdfff-4031-4a87-8a7a-1d05eae2a753","Type":"ContainerStarted","Data":"4e4dbfbeb6c72c3e4a3cda2a12d34614664b96a7e07a34114e53686df8aaf52d"}
Jul 28 15:16:34.011180 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.011159    2579 scope.go:117] "RemoveContainer" containerID="116bfb75b253a9b4c95dbdf00836d29a1fa04a34850718072107e3f47b6bc4ca"
Jul 28 15:16:34.011301 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.011164    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585417232"
Jul 28 15:16:34.514201 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.514149    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg" podUID="e5e4dc84-57f9-4032-8eb0-7e24ae94842a" containerName="main" containerID="cri-o://378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f" gracePeriod=45
Jul 28 15:16:34.770903 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.770838    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:16:34.861069 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-dshm\") pod \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") "
Jul 28 15:16:34.861274 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861082    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-home\") pod \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") "
Jul 28 15:16:34.861274 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861126    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tmp-dir\") pod \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") "
Jul 28 15:16:34.861274 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861149    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg2mw\" (UniqueName: \"kubernetes.io/projected/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-kube-api-access-gg2mw\") pod \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") "
Jul 28 15:16:34.861274 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-model-cache\") pod \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") "
Jul 28 15:16:34.861274 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tls-certs\") pod \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\" (UID: \"e5e4dc84-57f9-4032-8eb0-7e24ae94842a\") "
Jul 28 15:16:34.861582 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-home" (OuterVolumeSpecName: "home") pod "e5e4dc84-57f9-4032-8eb0-7e24ae94842a" (UID: "e5e4dc84-57f9-4032-8eb0-7e24ae94842a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:16:34.861582 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5e4dc84-57f9-4032-8eb0-7e24ae94842a" (UID: "e5e4dc84-57f9-4032-8eb0-7e24ae94842a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:16:34.861582 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5e4dc84-57f9-4032-8eb0-7e24ae94842a" (UID: "e5e4dc84-57f9-4032-8eb0-7e24ae94842a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:16:34.861686 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861591    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:34.861686 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.861612    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:34.863277 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.863254    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-dshm" (OuterVolumeSpecName: "dshm") pod "e5e4dc84-57f9-4032-8eb0-7e24ae94842a" (UID: "e5e4dc84-57f9-4032-8eb0-7e24ae94842a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:16:34.863356 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.863267    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5e4dc84-57f9-4032-8eb0-7e24ae94842a" (UID: "e5e4dc84-57f9-4032-8eb0-7e24ae94842a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:16:34.863356 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.863316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-kube-api-access-gg2mw" (OuterVolumeSpecName: "kube-api-access-gg2mw") pod "e5e4dc84-57f9-4032-8eb0-7e24ae94842a" (UID: "e5e4dc84-57f9-4032-8eb0-7e24ae94842a"). InnerVolumeSpecName "kube-api-access-gg2mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:16:34.962429 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.962351    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:34.962429 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.962424    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:34.962665 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.962439    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg2mw\" (UniqueName: \"kubernetes.io/projected/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-kube-api-access-gg2mw\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:34.962665 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:34.962469    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e4dc84-57f9-4032-8eb0-7e24ae94842a-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:35.019560 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.019524    2579 generic.go:356] "Generic (PLEG): container finished" podID="e5e4dc84-57f9-4032-8eb0-7e24ae94842a" containerID="378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f" exitCode=0
Jul 28 15:16:35.019737 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.019571    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"
Jul 28 15:16:35.019737 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.019617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg" event={"ID":"e5e4dc84-57f9-4032-8eb0-7e24ae94842a","Type":"ContainerDied","Data":"378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f"}
Jul 28 15:16:35.019737 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.019660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg" event={"ID":"e5e4dc84-57f9-4032-8eb0-7e24ae94842a","Type":"ContainerDied","Data":"44f2500698a387143d979e9e5453ddbc6b216ec13d2f06db5e186629d871dd0b"}
Jul 28 15:16:35.019737 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.019683    2579 scope.go:117] "RemoveContainer" containerID="378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f"
Jul 28 15:16:35.021415 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.021333    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f5fdfff-4031-4a87-8a7a-1d05eae2a753" containerID="3328aa33593c688ba50e63550903fc93b06369bb8a147905b80bd0fc6763eab6" exitCode=0
Jul 28 15:16:35.021415 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.021380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585418240" event={"ID":"8f5fdfff-4031-4a87-8a7a-1d05eae2a753","Type":"ContainerDied","Data":"3328aa33593c688ba50e63550903fc93b06369bb8a147905b80bd0fc6763eab6"}
Jul 28 15:16:35.031126 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.031103    2579 scope.go:117] "RemoveContainer" containerID="378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f"
Jul 28 15:16:35.031450 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:16:35.031429    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f\": container with ID starting with 378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f not found: ID does not exist" containerID="378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f"
Jul 28 15:16:35.031522 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.031458    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f"} err="failed to get container status \"378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f\": rpc error: code = NotFound desc = could not find container \"378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f\": container with ID starting with 378d77edeeb04d827b61640d78c8777985b7f6732ed23e6c5f730d8f1cd8b36f not found: ID does not exist"
Jul 28 15:16:35.047157 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.047125    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"]
Jul 28 15:16:35.050489 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.050456    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fd42ef/rate-test-model-fd42ef-kserve-5bcb59675f-ldzwg"]
Jul 28 15:16:35.252821 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.252780    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e4dc84-57f9-4032-8eb0-7e24ae94842a" path="/var/lib/kubelet/pods/e5e4dc84-57f9-4032-8eb0-7e24ae94842a/volumes"
Jul 28 15:16:35.491520 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.491372    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj" podUID="df727959-9df7-4879-870b-398ba98d7d7e" containerName="main" containerID="cri-o://06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55" gracePeriod=45
Jul 28 15:16:35.750643 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.750572    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:16:35.869576 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.869527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-tmp-dir\") pod \"df727959-9df7-4879-870b-398ba98d7d7e\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") "
Jul 28 15:16:35.869576 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.869576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df727959-9df7-4879-870b-398ba98d7d7e-tls-certs\") pod \"df727959-9df7-4879-870b-398ba98d7d7e\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") "
Jul 28 15:16:35.869863 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.869628    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b95vg\" (UniqueName: \"kubernetes.io/projected/df727959-9df7-4879-870b-398ba98d7d7e-kube-api-access-b95vg\") pod \"df727959-9df7-4879-870b-398ba98d7d7e\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") "
Jul 28 15:16:35.869863 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.869651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-home\") pod \"df727959-9df7-4879-870b-398ba98d7d7e\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") "
Jul 28 15:16:35.869863 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.869670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-model-cache\") pod \"df727959-9df7-4879-870b-398ba98d7d7e\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") "
Jul 28 15:16:35.869863 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.869691    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-dshm\") pod \"df727959-9df7-4879-870b-398ba98d7d7e\" (UID: \"df727959-9df7-4879-870b-398ba98d7d7e\") "
Jul 28 15:16:35.870148 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.869852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df727959-9df7-4879-870b-398ba98d7d7e" (UID: "df727959-9df7-4879-870b-398ba98d7d7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:16:35.870148 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.870005    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-home" (OuterVolumeSpecName: "home") pod "df727959-9df7-4879-870b-398ba98d7d7e" (UID: "df727959-9df7-4879-870b-398ba98d7d7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:16:35.870148 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.870015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "df727959-9df7-4879-870b-398ba98d7d7e" (UID: "df727959-9df7-4879-870b-398ba98d7d7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:16:35.870148 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.870019    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:35.871878 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.871852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-dshm" (OuterVolumeSpecName: "dshm") pod "df727959-9df7-4879-870b-398ba98d7d7e" (UID: "df727959-9df7-4879-870b-398ba98d7d7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:16:35.871985 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.871892    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df727959-9df7-4879-870b-398ba98d7d7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df727959-9df7-4879-870b-398ba98d7d7e" (UID: "df727959-9df7-4879-870b-398ba98d7d7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:16:35.871985 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.871896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df727959-9df7-4879-870b-398ba98d7d7e-kube-api-access-b95vg" (OuterVolumeSpecName: "kube-api-access-b95vg") pod "df727959-9df7-4879-870b-398ba98d7d7e" (UID: "df727959-9df7-4879-870b-398ba98d7d7e"). InnerVolumeSpecName "kube-api-access-b95vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:16:35.971235 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.971186    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df727959-9df7-4879-870b-398ba98d7d7e-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:35.971235 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.971225    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b95vg\" (UniqueName: \"kubernetes.io/projected/df727959-9df7-4879-870b-398ba98d7d7e-kube-api-access-b95vg\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:35.971235 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.971239    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:35.971548 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.971252    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:35.971548 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:35.971265    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df727959-9df7-4879-870b-398ba98d7d7e-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:36.029771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.029671    2579 generic.go:356] "Generic (PLEG): container finished" podID="df727959-9df7-4879-870b-398ba98d7d7e" containerID="06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55" exitCode=0
Jul 28 15:16:36.029771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.029724    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"
Jul 28 15:16:36.029771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.029755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj" event={"ID":"df727959-9df7-4879-870b-398ba98d7d7e","Type":"ContainerDied","Data":"06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55"}
Jul 28 15:16:36.030067 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.029798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj" event={"ID":"df727959-9df7-4879-870b-398ba98d7d7e","Type":"ContainerDied","Data":"c7bf0f6dabcd3a3f01ea45513cec699d965fd187b5a4190c70d30df58d2e653a"}
Jul 28 15:16:36.030067 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.029823    2579 scope.go:117] "RemoveContainer" containerID="06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55"
Jul 28 15:16:36.039680 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.039644    2579 scope.go:117] "RemoveContainer" containerID="06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55"
Jul 28 15:16:36.039988 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:16:36.039970    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55\": container with ID starting with 06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55 not found: ID does not exist" containerID="06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55"
Jul 28 15:16:36.040063 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.039994    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55"} err="failed to get container status \"06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55\": rpc error: code = NotFound desc = could not find container \"06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55\": container with ID starting with 06be8b6a9c64432a5ae0636bc30754880803c6a49ceb9d1bed3a77aacc2add55 not found: ID does not exist"
Jul 28 15:16:36.051031 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.050997    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"]
Jul 28 15:16:36.054637 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.054603    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc713f/rate-test-model-bc713f-kserve-5467588547-nbtdj"]
Jul 28 15:16:36.170304 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.170273    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:36.189967 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.189909    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7611-140335585418240_8f5fdfff-4031-4a87-8a7a-1d05eae2a753/test-curl-7611-140335585418240/0.log"
Jul 28 15:16:36.216412 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.216369    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:36.218188 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.218158    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:36.273942 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.273899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l8vp\" (UniqueName: \"kubernetes.io/projected/8f5fdfff-4031-4a87-8a7a-1d05eae2a753-kube-api-access-5l8vp\") pod \"8f5fdfff-4031-4a87-8a7a-1d05eae2a753\" (UID: \"8f5fdfff-4031-4a87-8a7a-1d05eae2a753\") "
Jul 28 15:16:36.276024 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.275993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5fdfff-4031-4a87-8a7a-1d05eae2a753-kube-api-access-5l8vp" (OuterVolumeSpecName: "kube-api-access-5l8vp") pod "8f5fdfff-4031-4a87-8a7a-1d05eae2a753" (UID: "8f5fdfff-4031-4a87-8a7a-1d05eae2a753"). InnerVolumeSpecName "kube-api-access-5l8vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.348416    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349223    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df727959-9df7-4879-870b-398ba98d7d7e" containerName="main"
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349243    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df727959-9df7-4879-870b-398ba98d7d7e" containerName="main"
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349283    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f5fdfff-4031-4a87-8a7a-1d05eae2a753" containerName="test-curl-7611-140335585418240"
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349293    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f5fdfff-4031-4a87-8a7a-1d05eae2a753" containerName="test-curl-7611-140335585418240"
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349321    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e4dc84-57f9-4032-8eb0-7e24ae94842a" containerName="main"
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349329    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e4dc84-57f9-4032-8eb0-7e24ae94842a" containerName="main"
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349501    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f5fdfff-4031-4a87-8a7a-1d05eae2a753" containerName="test-curl-7611-140335585418240"
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349522    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e4dc84-57f9-4032-8eb0-7e24ae94842a" containerName="main"
Jul 28 15:16:36.353636 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.349542    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df727959-9df7-4879-870b-398ba98d7d7e" containerName="main"
Jul 28 15:16:36.354946 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.354916    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:36.357591 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.357530    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:36.375351 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.375321    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l8vp\" (UniqueName: \"kubernetes.io/projected/8f5fdfff-4031-4a87-8a7a-1d05eae2a753-kube-api-access-5l8vp\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:36.475820 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.475777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7xh\" (UniqueName: \"kubernetes.io/projected/c4ecc400-0f85-4952-bc88-0f177330d1a3-kube-api-access-js7xh\") pod \"test-curl-7611-140335585418240\" (UID: \"c4ecc400-0f85-4952-bc88-0f177330d1a3\") " pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:36.577122 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.577080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7xh\" (UniqueName: \"kubernetes.io/projected/c4ecc400-0f85-4952-bc88-0f177330d1a3-kube-api-access-js7xh\") pod \"test-curl-7611-140335585418240\" (UID: \"c4ecc400-0f85-4952-bc88-0f177330d1a3\") " pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:36.585582 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.585544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7xh\" (UniqueName: \"kubernetes.io/projected/c4ecc400-0f85-4952-bc88-0f177330d1a3-kube-api-access-js7xh\") pod \"test-curl-7611-140335585418240\" (UID: \"c4ecc400-0f85-4952-bc88-0f177330d1a3\") " pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:36.665248 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.665206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:36.801038 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:36.801012    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:36.803769 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:16:36.803742    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4ecc400_0f85_4952_bc88_0f177330d1a3.slice/crio-723ae2fab66771bd1b18aab96ad73b9400a5726227d16fbd21de54dc1a27cdfd WatchSource:0}: Error finding container 723ae2fab66771bd1b18aab96ad73b9400a5726227d16fbd21de54dc1a27cdfd: Status 404 returned error can't find the container with id 723ae2fab66771bd1b18aab96ad73b9400a5726227d16fbd21de54dc1a27cdfd
Jul 28 15:16:37.036456 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:37.036418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585418240" event={"ID":"c4ecc400-0f85-4952-bc88-0f177330d1a3","Type":"ContainerStarted","Data":"723ae2fab66771bd1b18aab96ad73b9400a5726227d16fbd21de54dc1a27cdfd"}
Jul 28 15:16:37.038529 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:37.038504    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e4dbfbeb6c72c3e4a3cda2a12d34614664b96a7e07a34114e53686df8aaf52d"
Jul 28 15:16:37.038672 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:37.038547    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:37.050382 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:37.050350    2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7611-140335585418240" oldPodUID="8f5fdfff-4031-4a87-8a7a-1d05eae2a753" podUID="c4ecc400-0f85-4952-bc88-0f177330d1a3"
Jul 28 15:16:37.254727 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:37.254653    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f5fdfff-4031-4a87-8a7a-1d05eae2a753" path="/var/lib/kubelet/pods/8f5fdfff-4031-4a87-8a7a-1d05eae2a753/volumes"
Jul 28 15:16:37.255234 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:37.255218    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df727959-9df7-4879-870b-398ba98d7d7e" path="/var/lib/kubelet/pods/df727959-9df7-4879-870b-398ba98d7d7e/volumes"
Jul 28 15:16:38.045135 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:38.045099    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4ecc400-0f85-4952-bc88-0f177330d1a3" containerID="bffb8f1378dda6d2835f484aa1bdd723a925ad707339aad85862a7181fc7a9f0" exitCode=0
Jul 28 15:16:38.045539 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:38.045140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7611-140335585418240" event={"ID":"c4ecc400-0f85-4952-bc88-0f177330d1a3","Type":"ContainerDied","Data":"bffb8f1378dda6d2835f484aa1bdd723a925ad707339aad85862a7181fc7a9f0"}
Jul 28 15:16:39.189635 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:39.189603    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:39.212060 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:39.212035    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7611-140335585418240_c4ecc400-0f85-4952-bc88-0f177330d1a3/test-curl-7611-140335585418240/0.log"
Jul 28 15:16:39.234270 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:39.234227    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:39.239940 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:39.239910    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7611-140335585418240"]
Jul 28 15:16:39.301153 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:39.301047    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js7xh\" (UniqueName: \"kubernetes.io/projected/c4ecc400-0f85-4952-bc88-0f177330d1a3-kube-api-access-js7xh\") pod \"c4ecc400-0f85-4952-bc88-0f177330d1a3\" (UID: \"c4ecc400-0f85-4952-bc88-0f177330d1a3\") "
Jul 28 15:16:39.303270 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:39.303236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ecc400-0f85-4952-bc88-0f177330d1a3-kube-api-access-js7xh" (OuterVolumeSpecName: "kube-api-access-js7xh") pod "c4ecc400-0f85-4952-bc88-0f177330d1a3" (UID: "c4ecc400-0f85-4952-bc88-0f177330d1a3"). InnerVolumeSpecName "kube-api-access-js7xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:16:39.402349 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:39.402315    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7xh\" (UniqueName: \"kubernetes.io/projected/c4ecc400-0f85-4952-bc88-0f177330d1a3-kube-api-access-js7xh\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:16:39.800551 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:39.800512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:40.055971 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.055880    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7611-140335585418240"
Jul 28 15:16:40.055971 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.055909    2579 scope.go:117] "RemoveContainer" containerID="bffb8f1378dda6d2835f484aa1bdd723a925ad707339aad85862a7181fc7a9f0"
Jul 28 15:16:40.086603 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.086560    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"]
Jul 28 15:16:40.086986 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.086960    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ecc400-0f85-4952-bc88-0f177330d1a3" containerName="test-curl-7611-140335585418240"
Jul 28 15:16:40.086986 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.086985    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ecc400-0f85-4952-bc88-0f177330d1a3" containerName="test-curl-7611-140335585418240"
Jul 28 15:16:40.087163 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.087091    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ecc400-0f85-4952-bc88-0f177330d1a3" containerName="test-curl-7611-140335585418240"
Jul 28 15:16:40.091447 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.091353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.094920 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.094896    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-968c04-openshift-default-dockercfg-5mjks\""
Jul 28 15:16:40.100559 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.100451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"]
Jul 28 15:16:40.163351 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.163312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:40.208706 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.208667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-data\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.209106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.208713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cwn5\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-kube-api-access-7cwn5\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.209106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.208737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-envoy\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.209106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.208761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-token\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.209106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.208819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-socket\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.209106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.208844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-credential-socket\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.209106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.208868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-podinfo\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.209106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.208970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-certs\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.209106 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.209000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8726f4ca-469b-4882-8ad6-44c13cd4700c-istiod-ca-cert\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.309744 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-socket\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.309744 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-credential-socket\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.309744 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-podinfo\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.309744 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-certs\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310079 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8726f4ca-469b-4882-8ad6-44c13cd4700c-istiod-ca-cert\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310079 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-data\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310079 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cwn5\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-kube-api-access-7cwn5\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310079 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-envoy\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310079 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.309896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-token\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310331 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.310124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-socket\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310331 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.310142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-certs\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310331 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.310193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-credential-socket\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310468 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.310428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-data\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.310672 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.310652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8726f4ca-469b-4882-8ad6-44c13cd4700c-istiod-ca-cert\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.312262 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.312231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-envoy\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.312413 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.312377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-podinfo\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.324734 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.324704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-token\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.325053 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.325028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cwn5\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-kube-api-access-7cwn5\") pod \"e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") " pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.335875 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.335844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:40.405045 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.404998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:40.544819 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.544787    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"]
Jul 28 15:16:40.547136 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:16:40.547110    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8726f4ca_469b_4882_8ad6_44c13cd4700c.slice/crio-11f63862fe5996a320e2b19d46d01d8dffd6bcb28c4b35e76faa998dd0664ab9 WatchSource:0}: Error finding container 11f63862fe5996a320e2b19d46d01d8dffd6bcb28c4b35e76faa998dd0664ab9: Status 404 returned error can't find the container with id 11f63862fe5996a320e2b19d46d01d8dffd6bcb28c4b35e76faa998dd0664ab9
Jul 28 15:16:40.549454 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.549421    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:16:40.549557 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.549509    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:16:40.549557 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:40.549552    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:16:41.063879 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:41.063842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" event={"ID":"8726f4ca-469b-4882-8ad6-44c13cd4700c","Type":"ContainerStarted","Data":"b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e"}
Jul 28 15:16:41.064079 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:41.063903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" event={"ID":"8726f4ca-469b-4882-8ad6-44c13cd4700c","Type":"ContainerStarted","Data":"11f63862fe5996a320e2b19d46d01d8dffd6bcb28c4b35e76faa998dd0664ab9"}
Jul 28 15:16:41.082755 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:41.082692    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" podStartSLOduration=1.082673179 podStartE2EDuration="1.082673179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:16:41.080643653 +0000 UTC m=+3244.527493006" watchObservedRunningTime="2026-07-28 15:16:41.082673179 +0000 UTC m=+3244.529522488"
Jul 28 15:16:41.251692 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:41.251650    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4ecc400-0f85-4952-bc88-0f177330d1a3" path="/var/lib/kubelet/pods/c4ecc400-0f85-4952-bc88-0f177330d1a3/volumes"
Jul 28 15:16:41.406227 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:41.406127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:41.411647 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:41.411619    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:42.070320 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:42.070288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:42.071442 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:42.071416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:16:43.060053 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:43.060006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:43.161317 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:43.161281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:43.303295 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:43.303259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:43.562880 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:43.562848    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:43.929755 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:43.929651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:44.024411 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:44.024366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:16:51.779095 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:16:51.779060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:17:01.176836 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:01.176805    2579 generic.go:356] "Generic (PLEG): container finished" podID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerID="b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e" exitCode=0
Jul 28 15:17:01.177275 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:01.176845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" event={"ID":"8726f4ca-469b-4882-8ad6-44c13cd4700c","Type":"ContainerDied","Data":"b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e"}
Jul 28 15:17:01.177275 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:01.177138    2579 scope.go:117] "RemoveContainer" containerID="b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e"
Jul 28 15:17:01.178107 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:01.178078    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:01.178173 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:01.178147    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:01.178229 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:01.178218    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:02.183189 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:02.183155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" event={"ID":"8726f4ca-469b-4882-8ad6-44c13cd4700c","Type":"ContainerStarted","Data":"2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8"}
Jul 28 15:17:02.405187 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:02.405154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:17:02.405354 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:02.405289    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 28 15:17:02.405354 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:02.405333    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 28 15:17:03.405343 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:03.405311    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 28 15:17:03.405742 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:03.405370    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 28 15:17:04.405528 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:04.405498    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 28 15:17:04.405917 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:04.405547    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 28 15:17:05.405984 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:05.405949    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 28 15:17:05.406343 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:05.406004    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 28 15:17:06.409266 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:06.409228    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:17:06.409708 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:06.409689    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:17:06.410285 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:06.410267    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:17:08.244146 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:08.244117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:09.020650 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.020621    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:09.331526 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.331487    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"]
Jul 28 15:17:09.335113 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.335090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.337048 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.337022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86ef5d-openshift-default-dockercfg-j2qq4\""
Jul 28 15:17:09.346086 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.346065    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"]
Jul 28 15:17:09.471492 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c587f680-322a-4c78-b183-c6d97e058950-istio-podinfo\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.471662 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-istio-token\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.471662 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-socket\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.471662 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-envoy\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.471662 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-certs\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.471804 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-credential-socket\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.471804 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrc25\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-kube-api-access-qrc25\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.471804 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c587f680-322a-4c78-b183-c6d97e058950-istiod-ca-cert\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.471804 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.471798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-data\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.572710 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.572679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-data\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.572875 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.572722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c587f680-322a-4c78-b183-c6d97e058950-istio-podinfo\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.572875 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.572749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-istio-token\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.572875 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.572863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-socket\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573060 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.572912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-envoy\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573060 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.572944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-certs\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573060 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.572995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-credential-socket\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573060 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.573027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc25\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-kube-api-access-qrc25\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573255 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.573063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c587f680-322a-4c78-b183-c6d97e058950-istiod-ca-cert\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573255 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.573117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-data\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573255 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.573192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-socket\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573442 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.573355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-certs\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573442 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.573429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-credential-socket\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.573788 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.573767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c587f680-322a-4c78-b183-c6d97e058950-istiod-ca-cert\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.575043 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.575024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-envoy\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.575184 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.575167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c587f680-322a-4c78-b183-c6d97e058950-istio-podinfo\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.579594 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.579573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-istio-token\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.579753 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.579736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrc25\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-kube-api-access-qrc25\") pod \"e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") " pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.647480 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.647422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:09.717088 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.717054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:09.785956 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.785924    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"]
Jul 28 15:17:09.788582 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:17:09.788542    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc587f680_322a_4c78_b183_c6d97e058950.slice/crio-ae7cf51aa4f45ef76361a2c3b505e6e3d9a54dc7f43d65b1966068e4ee0f718a WatchSource:0}: Error finding container ae7cf51aa4f45ef76361a2c3b505e6e3d9a54dc7f43d65b1966068e4ee0f718a: Status 404 returned error can't find the container with id ae7cf51aa4f45ef76361a2c3b505e6e3d9a54dc7f43d65b1966068e4ee0f718a
Jul 28 15:17:09.791494 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.791459    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:09.791577 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.791537    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:09.791632 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.791574    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:09.821655 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:09.821619    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:10.222679 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:10.222642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" event={"ID":"c587f680-322a-4c78-b183-c6d97e058950","Type":"ContainerStarted","Data":"4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b"}
Jul 28 15:17:10.222868 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:10.222697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" event={"ID":"c587f680-322a-4c78-b183-c6d97e058950","Type":"ContainerStarted","Data":"ae7cf51aa4f45ef76361a2c3b505e6e3d9a54dc7f43d65b1966068e4ee0f718a"}
Jul 28 15:17:10.241618 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:10.241571    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" podStartSLOduration=1.241556599 podStartE2EDuration="1.241556599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:17:10.24000328 +0000 UTC m=+3273.686852589" watchObservedRunningTime="2026-07-28 15:17:10.241556599 +0000 UTC m=+3273.688405899"
Jul 28 15:17:10.648611 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:10.648580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:10.653294 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:10.653271    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:11.227745 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:11.227713    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:11.228789 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:11.228772    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:12.247791 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:12.247759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:12.324386 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:12.324353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:12.387838 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:12.387799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:12.815872 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:12.815838    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:12.944811 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:12.944778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:13.033379 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:13.033340    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:22.375357 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:22.375312    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:17:30.325966 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:30.325932    2579 generic.go:356] "Generic (PLEG): container finished" podID="c587f680-322a-4c78-b183-c6d97e058950" containerID="4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b" exitCode=0
Jul 28 15:17:30.326542 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:30.326008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" event={"ID":"c587f680-322a-4c78-b183-c6d97e058950","Type":"ContainerDied","Data":"4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b"}
Jul 28 15:17:30.326618 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:30.326588    2579 scope.go:117] "RemoveContainer" containerID="4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b"
Jul 28 15:17:30.327803 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:30.327776    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:30.327901 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:30.327841    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:30.327901 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:30.327875    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:17:31.333688 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:31.333649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" event={"ID":"c587f680-322a-4c78-b183-c6d97e058950","Type":"ContainerStarted","Data":"25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7"}
Jul 28 15:17:31.648227 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:31.648116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:31.648478 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:31.648455    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 15:17:31.648543 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:31.648504    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 15:17:32.648910 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:32.648866    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 15:17:32.649485 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:32.648942    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 15:17:33.648205 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:33.648141    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 15:17:33.648471 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:33.648228    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 15:17:34.648746 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:34.648685    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 15:17:34.649258 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:34.648773    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 15:17:35.648591 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:35.648542    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 28 15:17:35.648783 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:35.648613    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 28 15:17:36.652587 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:36.652545    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:36.653012 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:36.652859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:36.653582 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:36.653565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:17:37.367367 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:37.367332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:57.173755 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.173727    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"]
Jul 28 15:17:57.176267 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.176242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.179094 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.179073    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-968c04\"/\"default-dockercfg-tc8cp\""
Jul 28 15:17:57.179215 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.179113    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-968c04\"/\"kube-root-ca.crt\""
Jul 28 15:17:57.179215 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.179122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-968c04\"/\"ipp-route-968c04-kserve-self-signed-certs\""
Jul 28 15:17:57.179215 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.179188    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-968c04\"/\"openshift-service-ca.crt\""
Jul 28 15:17:57.186851 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.186823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"]
Jul 28 15:17:57.203104 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.203080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:17:57.251826 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.251795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqnc\" (UniqueName: \"kubernetes.io/projected/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-kube-api-access-glqnc\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.251826 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.251827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-home\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.252028 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.251846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tmp-dir\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.252028 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.251934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-dshm\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.252028 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.252006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-model-cache\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.252206 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.252052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tls-certs\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.352841 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.352814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-home\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.352998 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.352855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tmp-dir\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.352998 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.352891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-dshm\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.352998 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.352946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-model-cache\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.352998 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.352989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tls-certs\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.353215 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.353058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqnc\" (UniqueName: \"kubernetes.io/projected/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-kube-api-access-glqnc\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.353754 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.353730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-home\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.353954 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.353931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-model-cache\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.354363 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.354343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tmp-dir\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.356417 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.356361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-dshm\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.356829 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.356808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tls-certs\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.361188 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.361154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqnc\" (UniqueName: \"kubernetes.io/projected/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-kube-api-access-glqnc\") pod \"ipp-route-968c04-kserve-7bc746d7bc-zd4dq\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") " pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.487027 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.486961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:17:57.619946 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:57.619774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"]
Jul 28 15:17:57.622823 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:17:57.622792    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fec23c5_b2d8_4b4e_98b8_d9a982d90361.slice/crio-c768cd7c1b9b1de1cebfe0257f6766a56022097c371961c1cb245f7a7895e33b WatchSource:0}: Error finding container c768cd7c1b9b1de1cebfe0257f6766a56022097c371961c1cb245f7a7895e33b: Status 404 returned error can't find the container with id c768cd7c1b9b1de1cebfe0257f6766a56022097c371961c1cb245f7a7895e33b
Jul 28 15:17:58.473509 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:58.473472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq" event={"ID":"9fec23c5-b2d8-4b4e-98b8-d9a982d90361","Type":"ContainerStarted","Data":"d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b"}
Jul 28 15:17:58.473905 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:58.473521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq" event={"ID":"9fec23c5-b2d8-4b4e-98b8-d9a982d90361","Type":"ContainerStarted","Data":"c768cd7c1b9b1de1cebfe0257f6766a56022097c371961c1cb245f7a7895e33b"}
Jul 28 15:17:58.488962 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:17:58.488915    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq" podStartSLOduration=1.488899779 podStartE2EDuration="1.488899779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:17:58.48719827 +0000 UTC m=+3321.934047590" watchObservedRunningTime="2026-07-28 15:17:58.488899779 +0000 UTC m=+3321.935749087"
Jul 28 15:18:00.781562 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:00.781480    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:18:07.487232 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:07.487194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:18:07.487654 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:07.487253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:18:07.499766 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:07.499742    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:18:07.527311 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:07.527288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:18:30.178593 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:30.178549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:18:42.844299 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.844260    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"]
Jul 28 15:18:42.847664 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.847641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.849735 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.849713    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-adb651-openshift-default-dockercfg-m5r4n\""
Jul 28 15:18:42.859952 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.859925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-credential-socket\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.860070 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.859968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-certs\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.860129 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.860080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210c964c-d8c3-4799-8567-36715db464b8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.860178 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.860132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg85m\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-kube-api-access-pg85m\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.860178 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.860170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-istio-token\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.860269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.860197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-data\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.860269 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.860264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-envoy\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.860370 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.860289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210c964c-d8c3-4799-8567-36715db464b8-istio-podinfo\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.860370 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.860347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-socket\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.861262 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.861235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"]
Jul 28 15:18:42.961302 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-credential-socket\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961302 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-certs\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961600 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210c964c-d8c3-4799-8567-36715db464b8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961600 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg85m\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-kube-api-access-pg85m\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961600 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-istio-token\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961600 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-data\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961600 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-envoy\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961600 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210c964c-d8c3-4799-8567-36715db464b8-istio-podinfo\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961600 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-socket\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961948 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-credential-socket\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961948 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-data\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.961948 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-socket\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.962053 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.961945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-certs\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.962239 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.962215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210c964c-d8c3-4799-8567-36715db464b8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.963840 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.963819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-envoy\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.964057 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.964036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210c964c-d8c3-4799-8567-36715db464b8-istio-podinfo\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.968955 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.968931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg85m\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-kube-api-access-pg85m\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:42.969055 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:42.968974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-istio-token\") pod \"e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") " pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:43.157950 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.157859    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:43.298122 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.298096    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"]
Jul 28 15:18:43.299930 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:18:43.299898    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210c964c_d8c3_4799_8567_36715db464b8.slice/crio-6a0529d331faacf2261e99bbd81c335df40f3f3621a3a998ebdeb83710f8331b WatchSource:0}: Error finding container 6a0529d331faacf2261e99bbd81c335df40f3f3621a3a998ebdeb83710f8331b: Status 404 returned error can't find the container with id 6a0529d331faacf2261e99bbd81c335df40f3f3621a3a998ebdeb83710f8331b
Jul 28 15:18:43.302107 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.302072    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:18:43.302186 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.302143    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:18:43.302186 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.302170    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:18:43.533555 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.533475    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:18:43.631179 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.631144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:18:43.703693 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.703660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" event={"ID":"210c964c-d8c3-4799-8567-36715db464b8","Type":"ContainerStarted","Data":"da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d"}
Jul 28 15:18:43.703922 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.703719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" event={"ID":"210c964c-d8c3-4799-8567-36715db464b8","Type":"ContainerStarted","Data":"6a0529d331faacf2261e99bbd81c335df40f3f3621a3a998ebdeb83710f8331b"}
Jul 28 15:18:43.721760 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:43.721700    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" podStartSLOduration=1.721682646 podStartE2EDuration="1.721682646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:18:43.721058168 +0000 UTC m=+3367.167907479" watchObservedRunningTime="2026-07-28 15:18:43.721682646 +0000 UTC m=+3367.168531954"
Jul 28 15:18:44.158461 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:44.158428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:44.163555 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:44.163530    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:44.709174 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:44.709140    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:44.710241 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:44.710214    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:45.638011 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:45.637974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:18:45.716877 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:45.716844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:18:46.058296 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:46.058259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:18:46.181152 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:46.181120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:18:46.316149 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:46.316070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:18:46.419202 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:46.419160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:18:50.980440 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:50.980402    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:18:58.808050 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:58.808015    2579 generic.go:356] "Generic (PLEG): container finished" podID="210c964c-d8c3-4799-8567-36715db464b8" containerID="da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d" exitCode=0
Jul 28 15:18:58.808429 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:58.808062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" event={"ID":"210c964c-d8c3-4799-8567-36715db464b8","Type":"ContainerDied","Data":"da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d"}
Jul 28 15:18:58.808479 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:58.808443    2579 scope.go:117] "RemoveContainer" containerID="da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d"
Jul 28 15:18:58.809490 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:58.809457    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:18:58.809595 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:58.809545    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:18:58.809663 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:58.809592    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 15:18:59.709235 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:59.709205    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:18:59.815204 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:18:59.815175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" event={"ID":"210c964c-d8c3-4799-8567-36715db464b8","Type":"ContainerStarted","Data":"053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878"}
Jul 28 15:19:00.158948 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:00.158913    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:19:00.159810 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:00.159784    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:19:00.159935 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:00.159829    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 28 15:19:01.158408 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:01.158369    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:19:01.158860 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:01.158441    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 28 15:19:02.158888 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:02.158851    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:19:02.159280 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:02.158903    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 28 15:19:03.158411 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:03.158349    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:19:03.158602 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:03.158427    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 28 15:19:04.158682 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:04.158639    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Jul 28 15:19:04.159166 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:04.158713    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Jul 28 15:19:05.162619 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:05.162586    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:19:05.163098 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:05.162978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:19:05.163772 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:05.163751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:19:10.680509 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:10.680432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:16.255146 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:16.255113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:16.323350 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:16.323312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:16.404323 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:16.404293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:16.503094 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:16.503057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:16.815208 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:16.815175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:21.174191 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:21.174156    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:19:22.744617 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:22.744578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:22.845065 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:22.845029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:23.008173 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:23.008085    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"]
Jul 28 15:19:23.008349 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:23.008310    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy" containerID="cri-o://053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878" gracePeriod=30
Jul 28 15:19:23.030841 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:23.030813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:23.270862 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:23.270780    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:28.269388 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.269332    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:19:28.463604 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg85m\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-kube-api-access-pg85m\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.463604 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463611    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210c964c-d8c3-4799-8567-36715db464b8-istiod-ca-cert\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.463832 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463668    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-envoy\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.463832 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463693    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-certs\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.463832 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210c964c-d8c3-4799-8567-36715db464b8-istio-podinfo\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.463832 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-istio-token\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.463832 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-credential-socket\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.463832 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463825    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-data\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.464133 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-socket\") pod \"210c964c-d8c3-4799-8567-36715db464b8\" (UID: \"210c964c-d8c3-4799-8567-36715db464b8\") "
Jul 28 15:19:28.464133 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.463959    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:19:28.464236 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.464081    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210c964c-d8c3-4799-8567-36715db464b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:19:28.464236 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.464185    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:19:28.464236 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.464157    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.464440 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.464339    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:19:28.464848 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.464827    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:19:28.465968 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.465931    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:19:28.465968 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.465956    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/210c964c-d8c3-4799-8567-36715db464b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:19:28.466224 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.466207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:19:28.466289 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.466224    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-kube-api-access-pg85m" (OuterVolumeSpecName: "kube-api-access-pg85m") pod "210c964c-d8c3-4799-8567-36715db464b8" (UID: "210c964c-d8c3-4799-8567-36715db464b8"). InnerVolumeSpecName "kube-api-access-pg85m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:19:28.565283 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.565257    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210c964c-d8c3-4799-8567-36715db464b8-istio-podinfo\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.565283 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.565280    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-istio-token\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.565470 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.565289    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-credential-socket\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.565470 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.565298    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-data\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.565470 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.565308    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-workload-socket\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.565470 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.565317    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg85m\" (UniqueName: \"kubernetes.io/projected/210c964c-d8c3-4799-8567-36715db464b8-kube-api-access-pg85m\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.565470 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.565327    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210c964c-d8c3-4799-8567-36715db464b8-istiod-ca-cert\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.565470 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.565335    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210c964c-d8c3-4799-8567-36715db464b8-istio-envoy\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:19:28.965347 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.965270    2579 generic.go:356] "Generic (PLEG): container finished" podID="210c964c-d8c3-4799-8567-36715db464b8" containerID="053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878" exitCode=0
Jul 28 15:19:28.965347 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.965313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" event={"ID":"210c964c-d8c3-4799-8567-36715db464b8","Type":"ContainerDied","Data":"053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878"}
Jul 28 15:19:28.965347 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.965324    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"
Jul 28 15:19:28.965584 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.965366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt" event={"ID":"210c964c-d8c3-4799-8567-36715db464b8","Type":"ContainerDied","Data":"6a0529d331faacf2261e99bbd81c335df40f3f3621a3a998ebdeb83710f8331b"}
Jul 28 15:19:28.965584 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.965424    2579 scope.go:117] "RemoveContainer" containerID="053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878"
Jul 28 15:19:28.975104 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.975083    2579 scope.go:117] "RemoveContainer" containerID="da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d"
Jul 28 15:19:28.983188 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.983169    2579 scope.go:117] "RemoveContainer" containerID="053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878"
Jul 28 15:19:28.983534 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:19:28.983429    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878\": container with ID starting with 053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878 not found: ID does not exist" containerID="053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878"
Jul 28 15:19:28.983534 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.983464    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878"} err="failed to get container status \"053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878\": rpc error: code = NotFound desc = could not find container \"053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878\": container with ID starting with 053463ae8dad8032d2ccc64812b95cc5024379e2d8bedacd9ffbe0d678f82878 not found: ID does not exist"
Jul 28 15:19:28.983534 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.983487    2579 scope.go:117] "RemoveContainer" containerID="da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d"
Jul 28 15:19:28.983895 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:19:28.983866    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d\": container with ID starting with da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d not found: ID does not exist" containerID="da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d"
Jul 28 15:19:28.983976 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.983904    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d"} err="failed to get container status \"da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d\": rpc error: code = NotFound desc = could not find container \"da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d\": container with ID starting with da3ee6c31f838b0877fb79bbf8dfb1e52cdf12a2c0a4a86f198f40b57115a99d not found: ID does not exist"
Jul 28 15:19:28.986024 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.986002    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"]
Jul 28 15:19:28.991595 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:28.991574    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-adb651-openshift-default-fd749d46-sfpbt"]
Jul 28 15:19:29.258041 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:29.257965    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="210c964c-d8c3-4799-8567-36715db464b8" path="/var/lib/kubelet/pods/210c964c-d8c3-4799-8567-36715db464b8/volumes"
Jul 28 15:19:41.882243 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:41.882210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:19:42.706806 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:42.706770    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:42.772568 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:42.772541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:42.831176 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:42.831138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:42.888619 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:42.888574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:43.265688 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:43.265654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:48.246978 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:48.246941    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"]
Jul 28 15:19:48.294230 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:48.294197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:19:52.572219 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:19:52.572180    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:20:03.265904 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.265864    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq" podUID="9fec23c5-b2d8-4b4e-98b8-d9a982d90361" containerName="main" containerID="cri-o://d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b" gracePeriod=45
Jul 28 15:20:03.518186 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.518124    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:20:03.619471 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.619438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-model-cache\") pod \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") "
Jul 28 15:20:03.619668 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.619482    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tls-certs\") pod \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") "
Jul 28 15:20:03.619668 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.619566    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glqnc\" (UniqueName: \"kubernetes.io/projected/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-kube-api-access-glqnc\") pod \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") "
Jul 28 15:20:03.619813 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.619679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tmp-dir\") pod \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") "
Jul 28 15:20:03.619813 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.619760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-dshm\") pod \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") "
Jul 28 15:20:03.619813 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.619777    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fec23c5-b2d8-4b4e-98b8-d9a982d90361" (UID: "9fec23c5-b2d8-4b4e-98b8-d9a982d90361"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:03.619813 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.619795    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-home\") pod \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\" (UID: \"9fec23c5-b2d8-4b4e-98b8-d9a982d90361\") "
Jul 28 15:20:03.620017 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.619915    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fec23c5-b2d8-4b4e-98b8-d9a982d90361" (UID: "9fec23c5-b2d8-4b4e-98b8-d9a982d90361"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:03.620105 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.620083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-home" (OuterVolumeSpecName: "home") pod "9fec23c5-b2d8-4b4e-98b8-d9a982d90361" (UID: "9fec23c5-b2d8-4b4e-98b8-d9a982d90361"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:03.620157 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.620141    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-model-cache\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:03.620204 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.620160    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tmp-dir\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:03.621784 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.621756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fec23c5-b2d8-4b4e-98b8-d9a982d90361" (UID: "9fec23c5-b2d8-4b4e-98b8-d9a982d90361"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:20:03.621887 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.621782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-dshm" (OuterVolumeSpecName: "dshm") pod "9fec23c5-b2d8-4b4e-98b8-d9a982d90361" (UID: "9fec23c5-b2d8-4b4e-98b8-d9a982d90361"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:03.621887 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.621817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-kube-api-access-glqnc" (OuterVolumeSpecName: "kube-api-access-glqnc") pod "9fec23c5-b2d8-4b4e-98b8-d9a982d90361" (UID: "9fec23c5-b2d8-4b4e-98b8-d9a982d90361"). InnerVolumeSpecName "kube-api-access-glqnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:20:03.721564 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.721535    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glqnc\" (UniqueName: \"kubernetes.io/projected/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-kube-api-access-glqnc\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:03.721564 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.721559    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-dshm\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:03.721564 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.721569    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-home\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:03.721774 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:03.721577    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fec23c5-b2d8-4b4e-98b8-d9a982d90361-tls-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:04.138162 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.138125    2579 generic.go:356] "Generic (PLEG): container finished" podID="9fec23c5-b2d8-4b4e-98b8-d9a982d90361" containerID="d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b" exitCode=0
Jul 28 15:20:04.138367 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.138169    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"
Jul 28 15:20:04.138367 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.138210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq" event={"ID":"9fec23c5-b2d8-4b4e-98b8-d9a982d90361","Type":"ContainerDied","Data":"d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b"}
Jul 28 15:20:04.138367 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.138249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq" event={"ID":"9fec23c5-b2d8-4b4e-98b8-d9a982d90361","Type":"ContainerDied","Data":"c768cd7c1b9b1de1cebfe0257f6766a56022097c371961c1cb245f7a7895e33b"}
Jul 28 15:20:04.138367 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.138270    2579 scope.go:117] "RemoveContainer" containerID="d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b"
Jul 28 15:20:04.147544 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.147529    2579 scope.go:117] "RemoveContainer" containerID="d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b"
Jul 28 15:20:04.147796 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:20:04.147775    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b\": container with ID starting with d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b not found: ID does not exist" containerID="d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b"
Jul 28 15:20:04.147860 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.147804    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b"} err="failed to get container status \"d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b\": rpc error: code = NotFound desc = could not find container \"d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b\": container with ID starting with d792d5a302f44515100f3b24fc624f988f1e989a95319ec8f16455f4e3535d7b not found: ID does not exist"
Jul 28 15:20:04.157645 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.157623    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"]
Jul 28 15:20:04.166327 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:04.165553    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-968c04/ipp-route-968c04-kserve-7bc746d7bc-zd4dq"]
Jul 28 15:20:05.251822 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:05.251788    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fec23c5-b2d8-4b4e-98b8-d9a982d90361" path="/var/lib/kubelet/pods/9fec23c5-b2d8-4b4e-98b8-d9a982d90361/volumes"
Jul 28 15:20:24.695883 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:24.695846    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:25.624466 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:25.624432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:26.367890 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:26.367860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:31.568855 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:31.568817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:31.669317 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:31.669287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:31.854051 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:31.853962    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"]
Jul 28 15:20:31.854224 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:31.854202    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy" containerID="cri-o://2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8" gracePeriod=30
Jul 28 15:20:31.908271 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:31.908248    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:32.124038 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:32.123955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:36.412730 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:36.412680    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 15:20:36.413147 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:36.412737    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:20:37.106065 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.106042    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:20:37.173066 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.172993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-envoy\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173066 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173022    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8726f4ca-469b-4882-8ad6-44c13cd4700c-istiod-ca-cert\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173066 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-certs\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173268 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-token\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173268 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-podinfo\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173361 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173293    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-credential-socket\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173361 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-data\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173361 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173340    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:37.173361 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-socket\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173607 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cwn5\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-kube-api-access-7cwn5\") pod \"8726f4ca-469b-4882-8ad6-44c13cd4700c\" (UID: \"8726f4ca-469b-4882-8ad6-44c13cd4700c\") "
Jul 28 15:20:37.173668 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173414    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8726f4ca-469b-4882-8ad6-44c13cd4700c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:20:37.173737 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:37.173795 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:37.173867 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173797    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8726f4ca-469b-4882-8ad6-44c13cd4700c-istiod-ca-cert\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.173867 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173819    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.173867 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.173832    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-credential-socket\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.174202 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.174181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-data" (OuterVolumeSpecName: "istio-data") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:37.175302 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.175279    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-token" (OuterVolumeSpecName: "istio-token") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:20:37.175443 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.175323    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:20:37.175615 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.175597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:37.175700 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.175665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-kube-api-access-7cwn5" (OuterVolumeSpecName: "kube-api-access-7cwn5") pod "8726f4ca-469b-4882-8ad6-44c13cd4700c" (UID: "8726f4ca-469b-4882-8ad6-44c13cd4700c"). InnerVolumeSpecName "kube-api-access-7cwn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:20:37.275185 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.275160    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-token\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.275185 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.275181    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-podinfo\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.275307 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.275190    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-data\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.275307 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.275199    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-workload-socket\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.275307 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.275207    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cwn5\" (UniqueName: \"kubernetes.io/projected/8726f4ca-469b-4882-8ad6-44c13cd4700c-kube-api-access-7cwn5\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.275307 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.275215    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8726f4ca-469b-4882-8ad6-44c13cd4700c-istio-envoy\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:37.306432 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.306405    2579 generic.go:356] "Generic (PLEG): container finished" podID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerID="2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8" exitCode=0
Jul 28 15:20:37.306528 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.306444    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"
Jul 28 15:20:37.306528 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.306461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" event={"ID":"8726f4ca-469b-4882-8ad6-44c13cd4700c","Type":"ContainerDied","Data":"2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8"}
Jul 28 15:20:37.306528 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.306510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9" event={"ID":"8726f4ca-469b-4882-8ad6-44c13cd4700c","Type":"ContainerDied","Data":"11f63862fe5996a320e2b19d46d01d8dffd6bcb28c4b35e76faa998dd0664ab9"}
Jul 28 15:20:37.306635 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.306537    2579 scope.go:117] "RemoveContainer" containerID="2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8"
Jul 28 15:20:37.316614 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.316581    2579 scope.go:117] "RemoveContainer" containerID="b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e"
Jul 28 15:20:37.321429 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.321405    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"]
Jul 28 15:20:37.324121 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.324103    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-968c04-openshift-default-59cdc8ff94-kprj9"]
Jul 28 15:20:37.324994 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.324980    2579 scope.go:117] "RemoveContainer" containerID="2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8"
Jul 28 15:20:37.325214 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:20:37.325195    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8\": container with ID starting with 2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8 not found: ID does not exist" containerID="2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8"
Jul 28 15:20:37.325284 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.325224    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8"} err="failed to get container status \"2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8\": rpc error: code = NotFound desc = could not find container \"2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8\": container with ID starting with 2e7797240181d98330e8707596c2ec6bd436d7885c25726e54509df479995af8 not found: ID does not exist"
Jul 28 15:20:37.325284 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.325249    2579 scope.go:117] "RemoveContainer" containerID="b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e"
Jul 28 15:20:37.325541 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:20:37.325523    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e\": container with ID starting with b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e not found: ID does not exist" containerID="b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e"
Jul 28 15:20:37.325593 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:37.325550    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e"} err="failed to get container status \"b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e\": rpc error: code = NotFound desc = could not find container \"b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e\": container with ID starting with b2f80c05e7a7d25c664eca997e9594c54e5c3f104bee766a01ac78c0635cc45e not found: ID does not exist"
Jul 28 15:20:39.252509 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:39.252467    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" path="/var/lib/kubelet/pods/8726f4ca-469b-4882-8ad6-44c13cd4700c/volumes"
Jul 28 15:20:42.634256 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:42.634219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:42.734076 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:42.734043    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:42.833082 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:42.833036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:43.157101 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:43.157069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:45.976482 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:45.976448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:20:49.310362 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:49.310328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:49.411144 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:49.411112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:49.552101 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:49.552065    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"]
Jul 28 15:20:49.552300 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:49.552273    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy" containerID="cri-o://25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7" gracePeriod=30
Jul 28 15:20:49.578741 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:49.578674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:49.808118 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:49.808087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:20:51.655612 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:51.655579    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 15:20:51.655964 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:51.655641    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:20:54.811594 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.811569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:20:54.907855 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.907783    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-socket\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.907855 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.907823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c587f680-322a-4c78-b183-c6d97e058950-istiod-ca-cert\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.907855 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.907856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-certs\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.908100 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.907882    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrc25\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-kube-api-access-qrc25\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.908100 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.907904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-envoy\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.908100 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.907935    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c587f680-322a-4c78-b183-c6d97e058950-istio-podinfo\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.908100 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.907960    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-data\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.908100 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.908037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-istio-token\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.908100 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.908063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-credential-socket\") pod \"c587f680-322a-4c78-b183-c6d97e058950\" (UID: \"c587f680-322a-4c78-b183-c6d97e058950\") "
Jul 28 15:20:54.908100 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.908064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:54.908493 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.908212    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c587f680-322a-4c78-b183-c6d97e058950-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:20:54.908493 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.908302    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-socket\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:54.908493 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.908484    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:54.908666 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.908550    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:54.908966 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.908940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-data" (OuterVolumeSpecName: "istio-data") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:54.910054 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.910030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-kube-api-access-qrc25" (OuterVolumeSpecName: "kube-api-access-qrc25") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "kube-api-access-qrc25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:20:54.910157 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.910061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:20:54.910376 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.910355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c587f680-322a-4c78-b183-c6d97e058950-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 15:20:54.910463 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:54.910371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-istio-token" (OuterVolumeSpecName: "istio-token") pod "c587f680-322a-4c78-b183-c6d97e058950" (UID: "c587f680-322a-4c78-b183-c6d97e058950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:20:55.009608 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.009582    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrc25\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-kube-api-access-qrc25\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:55.009608 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.009605    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-envoy\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:55.009747 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.009614    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c587f680-322a-4c78-b183-c6d97e058950-istio-podinfo\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:55.009747 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.009623    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-istio-data\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:55.009747 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.009631    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c587f680-322a-4c78-b183-c6d97e058950-istio-token\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:55.009747 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.009639    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-credential-socket\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:55.009747 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.009647    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c587f680-322a-4c78-b183-c6d97e058950-istiod-ca-cert\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:55.009747 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.009654    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c587f680-322a-4c78-b183-c6d97e058950-workload-certs\") on node \"ip-10-0-141-160.ec2.internal\" DevicePath \"\""
Jul 28 15:20:55.391169 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.391135    2579 generic.go:356] "Generic (PLEG): container finished" podID="c587f680-322a-4c78-b183-c6d97e058950" containerID="25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7" exitCode=0
Jul 28 15:20:55.391284 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.391177    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"
Jul 28 15:20:55.391284 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.391196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" event={"ID":"c587f680-322a-4c78-b183-c6d97e058950","Type":"ContainerDied","Data":"25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7"}
Jul 28 15:20:55.391284 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.391239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj" event={"ID":"c587f680-322a-4c78-b183-c6d97e058950","Type":"ContainerDied","Data":"ae7cf51aa4f45ef76361a2c3b505e6e3d9a54dc7f43d65b1966068e4ee0f718a"}
Jul 28 15:20:55.391970 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.391264    2579 scope.go:117] "RemoveContainer" containerID="25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7"
Jul 28 15:20:55.400847 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.400828    2579 scope.go:117] "RemoveContainer" containerID="4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b"
Jul 28 15:20:55.407042 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.407020    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"]
Jul 28 15:20:55.409657 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.409597    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86ef5d-openshift-default-8599459c84-62dzj"]
Jul 28 15:20:55.409707 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.409683    2579 scope.go:117] "RemoveContainer" containerID="25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7"
Jul 28 15:20:55.409959 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:20:55.409937    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7\": container with ID starting with 25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7 not found: ID does not exist" containerID="25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7"
Jul 28 15:20:55.410015 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.409968    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7"} err="failed to get container status \"25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7\": rpc error: code = NotFound desc = could not find container \"25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7\": container with ID starting with 25b13a7bed2ffed7269bd0e0a0cd006987671f08f12c0f50415bd83a1a729ab7 not found: ID does not exist"
Jul 28 15:20:55.410015 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.409985    2579 scope.go:117] "RemoveContainer" containerID="4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b"
Jul 28 15:20:55.410227 ip-10-0-141-160 kubenswrapper[2579]: E0728 15:20:55.410208    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b\": container with ID starting with 4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b not found: ID does not exist" containerID="4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b"
Jul 28 15:20:55.410287 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:55.410238    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b"} err="failed to get container status \"4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b\": rpc error: code = NotFound desc = could not find container \"4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b\": container with ID starting with 4afae4e457ddbccd3df64cbcbe7c9338b001e9a21b325795a003b98cfa3ca22b not found: ID does not exist"
Jul 28 15:20:57.253044 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:20:57.253005    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c587f680-322a-4c78-b183-c6d97e058950" path="/var/lib/kubelet/pods/c587f680-322a-4c78-b183-c6d97e058950/volumes"
Jul 28 15:21:00.342653 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:00.342533    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"]
Jul 28 15:21:00.345064 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:00.345034    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fd42ef-29754195-l8bst"]
Jul 28 15:21:00.346559 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:00.346529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:00.445273 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:00.445246    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:00.533946 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:00.533912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:00.873581 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:00.873547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:01.251552 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:01.251470    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dfb0510-1b88-49c7-b9d8-3568dc21fcf6" path="/var/lib/kubelet/pods/0dfb0510-1b88-49c7-b9d8-3568dc21fcf6/volumes"
Jul 28 15:21:03.870714 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:03.870681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:21:05.971491 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:05.971459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:07.115840 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:07.115807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:07.175011 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:07.174979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:07.376880 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:07.376803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:08.418359 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:08.418327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:18.112324 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:18.112289    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"]
Jul 28 15:21:18.113889 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:18.113864    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc713f-29754195-xlwn8"]
Jul 28 15:21:18.131333 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:18.131302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:18.265913 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:18.265878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:18.379287 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:18.379205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:18.672575 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:18.672491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:19.251786 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:19.251754    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3cd9a00-8f67-4653-a86a-b6392e8f644c" path="/var/lib/kubelet/pods/d3cd9a00-8f67-4653-a86a-b6392e8f644c/volumes"
Jul 28 15:21:20.971105 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:20.971072    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dlp89"]
Jul 28 15:21:25.034430 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:25.034370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:25.137596 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:25.137565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:25.299897 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:25.299851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:25.525716 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:25.525678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:25.929900 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:25.929863    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-545c77995f-nfg54_40a8465c-258b-4f2d-803b-d9526ae00c95/authorino/0.log"
Jul 28 15:21:31.388131 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:31.388090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5nmq6_8cbe6dc3-8ae8-408c-bfba-0a50737c5708/manager/0.log"
Jul 28 15:21:31.511167 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:31.511133    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5zcc4_b0574a69-64d6-472c-88cf-e616bb771f9f/manager/0.log"
Jul 28 15:21:31.769205 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:31.769127    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-d4bl7_6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127/server/0.log"
Jul 28 15:21:31.893019 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:31.892988    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rlsz7_1c75c367-3583-405a-a265-f8dcdde207aa/manager/0.log"
Jul 28 15:21:32.152285 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:32.152257    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f457fb848-fzzdw_3ba14e51-17c9-4e46-937a-a63acea89d7e/copy-manifests/0.log"
Jul 28 15:21:32.164990 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:32.164964    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f457fb848-fzzdw_3ba14e51-17c9-4e46-937a-a63acea89d7e/manager/0.log"
Jul 28 15:21:32.772958 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:32.772929    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754165-r8b6q_6a6b7a40-97f7-4bd6-a604-ecb6e8e9ec5f/cleanup/0.log"
Jul 28 15:21:34.412962 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:34.412903    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-545c77995f-nfg54_40a8465c-258b-4f2d-803b-d9526ae00c95/authorino/0.log"
Jul 28 15:21:34.535173 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:34.535114    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vwjzn_606d89cc-4b15-4a9d-8d0d-8c3af9006b2e/manager/0.log"
Jul 28 15:21:34.647758 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:34.647731    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-svjl9_42a9eea7-1c10-4b86-8f1a-f06ed03800b3/manager/0.log"
Jul 28 15:21:34.886603 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:34.886577    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fxk7p_c29431f9-e60b-4c91-b9ef-77000f3ce73c/registry-server/0.log"
Jul 28 15:21:35.007766 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:35.007742    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-q59lf_2ef284aa-5f29-41ae-b720-eea7dc6a6c8d/manager/0.log"
Jul 28 15:21:35.120095 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:35.120072    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dlp89_2ec37ec5-3979-45d3-8cf6-83c9ba26ff96/limitador/0.log"
Jul 28 15:21:35.239908 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:35.239849    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-j7psw_ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2/manager/0.log"
Jul 28 15:21:35.838943 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:35.838919    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f6cf6d8-2f9gn_a84f130a-8c89-4ebc-8088-a2ea767a4ff9/kube-auth-proxy/0.log"
Jul 28 15:21:36.071500 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:36.071476    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gxhkr_9e868fe7-3c75-486d-8632-9b2840bdbf17/istio-proxy/1.log"
Jul 28 15:21:37.100075 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:37.100045    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhh9ms_d4d80999-e812-4174-935a-8e5ee6639102/main/0.log"
Jul 28 15:21:37.327778 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:37.327749    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5tc4m_f6a62f39-1150-4205-a3d2-b8bc079587c7/main/0.log"
Jul 28 15:21:37.483595 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:37.483503    2579 scope.go:117] "RemoveContainer" containerID="f2be4c30a15988b95eaf6b188b187da9e289f08e4eaa1e146c72d27337dcfb47"
Jul 28 15:21:37.491313 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:37.491290    2579 scope.go:117] "RemoveContainer" containerID="54da89c32dcb00e623a1e0c19539a93fb37439e228c76be519ef37e4d41acf49"
Jul 28 15:21:43.227429 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:43.227371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:43.347776 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:43.347745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log" limit=1024
Jul 28 15:21:46.541460 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:46.541428    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-txplh_a00f8f97-a454-45d2-9758-e09e9f0ff6f3/global-pull-secret-syncer/0.log"
Jul 28 15:21:46.589047 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:46.589011    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-975st_870f6c21-2fc5-436b-9f77-77b53329ca23/konnectivity-agent/0.log"
Jul 28 15:21:46.706180 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:46.706154    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-160.ec2.internal_1a6857bda9fd953712469ebe65794e79/haproxy/0.log"
Jul 28 15:21:51.200208 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:51.200177    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-545c77995f-nfg54_40a8465c-258b-4f2d-803b-d9526ae00c95/authorino/0.log"
Jul 28 15:21:51.224643 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:51.224617    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vwjzn_606d89cc-4b15-4a9d-8d0d-8c3af9006b2e/manager/0.log"
Jul 28 15:21:51.263243 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:51.263222    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-svjl9_42a9eea7-1c10-4b86-8f1a-f06ed03800b3/manager/0.log"
Jul 28 15:21:51.336370 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:51.336345    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fxk7p_c29431f9-e60b-4c91-b9ef-77000f3ce73c/registry-server/0.log"
Jul 28 15:21:51.491799 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:51.491719    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-q59lf_2ef284aa-5f29-41ae-b720-eea7dc6a6c8d/manager/0.log"
Jul 28 15:21:51.523164 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:51.523143    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dlp89_2ec37ec5-3979-45d3-8cf6-83c9ba26ff96/limitador/0.log"
Jul 28 15:21:51.601285 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:51.601262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-j7psw_ac04e0bc-ec1f-4e99-b31c-9eb9f875aae2/manager/0.log"
Jul 28 15:21:53.543735 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:53.543707    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lftxv_6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1/node-exporter/0.log"
Jul 28 15:21:53.568422 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:53.568387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lftxv_6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1/kube-rbac-proxy/0.log"
Jul 28 15:21:53.587795 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:53.587778    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lftxv_6b5a8157-ece4-4714-a9c6-bb95c8b2b5e1/init-textfile/0.log"
Jul 28 15:21:57.744996 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.744963    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"]
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745278    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745289    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745299    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745304    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745311    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745317    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745324    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745330    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745336    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745341    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745358    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fec23c5-b2d8-4b4e-98b8-d9a982d90361" containerName="main"
Jul 28 15:21:57.745374 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745363    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fec23c5-b2d8-4b4e-98b8-d9a982d90361" containerName="main"
Jul 28 15:21:57.745771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745423    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c587f680-322a-4c78-b183-c6d97e058950" containerName="istio-proxy"
Jul 28 15:21:57.745771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745432    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy"
Jul 28 15:21:57.745771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745440    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fec23c5-b2d8-4b4e-98b8-d9a982d90361" containerName="main"
Jul 28 15:21:57.745771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745445    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy"
Jul 28 15:21:57.745771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745450    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="210c964c-d8c3-4799-8567-36715db464b8" containerName="istio-proxy"
Jul 28 15:21:57.745771 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.745454    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8726f4ca-469b-4882-8ad6-44c13cd4700c" containerName="istio-proxy"
Jul 28 15:21:57.748338 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.748317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.750277 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.750253    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9w7l5\"/\"kube-root-ca.crt\""
Jul 28 15:21:57.750277 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.750270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9w7l5\"/\"openshift-service-ca.crt\""
Jul 28 15:21:57.750450 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.750270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9w7l5\"/\"default-dockercfg-krzlf\""
Jul 28 15:21:57.757241 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.757220    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"]
Jul 28 15:21:57.858207 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.858185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-proc\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.858290 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.858219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-podres\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.858290 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.858238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdwq\" (UniqueName: \"kubernetes.io/projected/1e258dd6-8b5a-440f-afd6-aa0009e9439b-kube-api-access-dtdwq\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.858364 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.858299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-sys\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.858364 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.858350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-lib-modules\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959439 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-sys\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959508 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-lib-modules\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959553 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-proc\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959553 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-sys\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959553 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-podres\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959653 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdwq\" (UniqueName: \"kubernetes.io/projected/1e258dd6-8b5a-440f-afd6-aa0009e9439b-kube-api-access-dtdwq\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959653 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-proc\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959653 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-lib-modules\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.959782 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.959680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e258dd6-8b5a-440f-afd6-aa0009e9439b-podres\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:57.966421 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:57.966382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtdwq\" (UniqueName: \"kubernetes.io/projected/1e258dd6-8b5a-440f-afd6-aa0009e9439b-kube-api-access-dtdwq\") pod \"perf-node-gather-daemonset-mlkwh\" (UID: \"1e258dd6-8b5a-440f-afd6-aa0009e9439b\") " pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:58.058205 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:58.058187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:58.180696 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:58.180659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"]
Jul 28 15:21:58.182969 ip-10-0-141-160 kubenswrapper[2579]: W0728 15:21:58.182941    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e258dd6_8b5a_440f_afd6_aa0009e9439b.slice/crio-47c9fcf9c4f017994aa54aafb5b022daa25b08c250e36db991856d7967d56450 WatchSource:0}: Error finding container 47c9fcf9c4f017994aa54aafb5b022daa25b08c250e36db991856d7967d56450: Status 404 returned error can't find the container with id 47c9fcf9c4f017994aa54aafb5b022daa25b08c250e36db991856d7967d56450
Jul 28 15:21:58.688068 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:58.688030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh" event={"ID":"1e258dd6-8b5a-440f-afd6-aa0009e9439b","Type":"ContainerStarted","Data":"305aa22ba2015435c6771f0201aef1e2d2986536cdf7fb0e447df5bbe49c7fca"}
Jul 28 15:21:58.688205 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:58.688097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh" event={"ID":"1e258dd6-8b5a-440f-afd6-aa0009e9439b","Type":"ContainerStarted","Data":"47c9fcf9c4f017994aa54aafb5b022daa25b08c250e36db991856d7967d56450"}
Jul 28 15:21:58.688205 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:58.688124    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:21:58.703033 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:58.702993    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh" podStartSLOduration=1.702979285 podStartE2EDuration="1.702979285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:21:58.70222207 +0000 UTC m=+3562.149071377" watchObservedRunningTime="2026-07-28 15:21:58.702979285 +0000 UTC m=+3562.149828593"
Jul 28 15:21:58.943987 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:58.943935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-px49m_904fca85-96bf-4856-b032-c57d0ab7cfbc/dns/0.log"
Jul 28 15:21:58.962116 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:58.962094    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-px49m_904fca85-96bf-4856-b032-c57d0ab7cfbc/kube-rbac-proxy/0.log"
Jul 28 15:21:59.032276 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:59.032253    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nxbfp_5edec3b9-e837-4bb1-9ec5-ffe2163f36df/dns-node-resolver/0.log"
Jul 28 15:21:59.622539 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:21:59.622520    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wrgjj_6921424a-8439-4602-acc4-e525c0779616/node-ca/0.log"
Jul 28 15:22:00.725896 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:00.725868    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f6cf6d8-2f9gn_a84f130a-8c89-4ebc-8088-a2ea767a4ff9/kube-auth-proxy/0.log"
Jul 28 15:22:00.774667 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:00.774643    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gxhkr_9e868fe7-3c75-486d-8632-9b2840bdbf17/istio-proxy/0.log"
Jul 28 15:22:00.799983 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:00.799957    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gxhkr_9e868fe7-3c75-486d-8632-9b2840bdbf17/istio-proxy/1.log"
Jul 28 15:22:01.706626 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:01.706600    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kjgps_5979f987-8fc4-42fc-9729-6db64e0c42bd/serve-healthcheck-canary/0.log"
Jul 28 15:22:02.206997 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:02.206977    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmzdr_9d84cfec-3afe-40e2-a59a-1a35e6b09523/kube-rbac-proxy/0.log"
Jul 28 15:22:02.225630 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:02.225609    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmzdr_9d84cfec-3afe-40e2-a59a-1a35e6b09523/exporter/0.log"
Jul 28 15:22:02.245521 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:02.245499    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmzdr_9d84cfec-3afe-40e2-a59a-1a35e6b09523/extractor/0.log"
Jul 28 15:22:04.471182 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:04.471158    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5nmq6_8cbe6dc3-8ae8-408c-bfba-0a50737c5708/manager/0.log"
Jul 28 15:22:04.676872 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:04.676845    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5zcc4_b0574a69-64d6-472c-88cf-e616bb771f9f/manager/0.log"
Jul 28 15:22:04.701944 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:04.701913    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9w7l5/perf-node-gather-daemonset-mlkwh"
Jul 28 15:22:04.883169 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:04.883142    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-d4bl7_6a0e5bfb-9881-4d8e-a8d2-4b1a7d9c9127/server/0.log"
Jul 28 15:22:04.952503 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:04.952483    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rlsz7_1c75c367-3583-405a-a265-f8dcdde207aa/manager/0.log"
Jul 28 15:22:05.075104 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:05.075080    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f457fb848-fzzdw_3ba14e51-17c9-4e46-937a-a63acea89d7e/manager/0.log"
Jul 28 15:22:05.090098 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:05.090077    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f457fb848-fzzdw_3ba14e51-17c9-4e46-937a-a63acea89d7e/copy-manifests/0.log"
Jul 28 15:22:07.249850 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:07.249783    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f6cf6d8-2f9gn_a84f130a-8c89-4ebc-8088-a2ea767a4ff9/kube-auth-proxy/0.log"
Jul 28 15:22:07.286907 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:07.286885    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gxhkr_9e868fe7-3c75-486d-8632-9b2840bdbf17/istio-proxy/0.log"
Jul 28 15:22:07.313398 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:07.313376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gxhkr_9e868fe7-3c75-486d-8632-9b2840bdbf17/istio-proxy/1.log"
Jul 28 15:22:08.972091 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:08.972012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f6cf6d8-2f9gn_a84f130a-8c89-4ebc-8088-a2ea767a4ff9/kube-auth-proxy/0.log"
Jul 28 15:22:09.006867 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:09.006841    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gxhkr_9e868fe7-3c75-486d-8632-9b2840bdbf17/istio-proxy/0.log"
Jul 28 15:22:09.034552 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:09.034528    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gxhkr_9e868fe7-3c75-486d-8632-9b2840bdbf17/istio-proxy/1.log"
Jul 28 15:22:13.300332 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.300300    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sczc4_4ca104f8-d1bd-4ea6-b807-b05829d737fe/kube-multus-additional-cni-plugins/0.log"
Jul 28 15:22:13.318920 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.318895    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sczc4_4ca104f8-d1bd-4ea6-b807-b05829d737fe/egress-router-binary-copy/0.log"
Jul 28 15:22:13.338144 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.338126    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sczc4_4ca104f8-d1bd-4ea6-b807-b05829d737fe/cni-plugins/0.log"
Jul 28 15:22:13.356878 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.356859    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sczc4_4ca104f8-d1bd-4ea6-b807-b05829d737fe/bond-cni-plugin/0.log"
Jul 28 15:22:13.376000 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.375980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sczc4_4ca104f8-d1bd-4ea6-b807-b05829d737fe/routeoverride-cni/0.log"
Jul 28 15:22:13.394811 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.394796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sczc4_4ca104f8-d1bd-4ea6-b807-b05829d737fe/whereabouts-cni-bincopy/0.log"
Jul 28 15:22:13.413758 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.413738    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sczc4_4ca104f8-d1bd-4ea6-b807-b05829d737fe/whereabouts-cni/0.log"
Jul 28 15:22:13.593626 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.593606    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l6zvc_dcaae002-b584-4e0a-b423-86454a1a68cc/kube-multus/0.log"
Jul 28 15:22:13.692404 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.692370    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h5w88_ed53709b-b3b8-4ef7-a9c2-35a9789dac55/network-metrics-daemon/0.log"
Jul 28 15:22:13.708109 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:13.708085    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h5w88_ed53709b-b3b8-4ef7-a9c2-35a9789dac55/kube-rbac-proxy/0.log"
Jul 28 15:22:14.945499 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:14.945469    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xvvg4_bf76e089-703b-4a88-ab06-e23f370322bf/ovn-controller/0.log"
Jul 28 15:22:14.963118 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:14.963099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xvvg4_bf76e089-703b-4a88-ab06-e23f370322bf/ovn-acl-logging/0.log"
Jul 28 15:22:14.984507 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:14.984490    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xvvg4_bf76e089-703b-4a88-ab06-e23f370322bf/kube-rbac-proxy-node/0.log"
Jul 28 15:22:15.004205 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:15.004187    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xvvg4_bf76e089-703b-4a88-ab06-e23f370322bf/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 15:22:15.022331 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:15.022314    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xvvg4_bf76e089-703b-4a88-ab06-e23f370322bf/northd/0.log"
Jul 28 15:22:15.041832 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:15.041816    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xvvg4_bf76e089-703b-4a88-ab06-e23f370322bf/nbdb/0.log"
Jul 28 15:22:15.060381 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:15.060366    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xvvg4_bf76e089-703b-4a88-ab06-e23f370322bf/sbdb/0.log"
Jul 28 15:22:15.114958 ip-10-0-141-160 kubenswrapper[2579]: I0728 15:22:15.114935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xvvg4_bf76e089-703b-4a88-ab06-e23f370322bf/ovnkube-controller/0.log"