Jul 06 19:04:02.061459 ip-10-0-131-1 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 19:04:02.061470 ip-10-0-131-1 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 19:04:02.061477 ip-10-0-131-1 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 19:04:02.061738 ip-10-0-131-1 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 19:04:12.088029 ip-10-0-131-1 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 19:04:12.088051 ip-10-0-131-1 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1df260641a1a4cc19149f5b3742fa5fa -- Jul 06 19:06:10.693069 ip-10-0-131-1 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:06:11.098693 ip-10-0-131-1 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:06:11.098693 ip-10-0-131-1 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:06:11.098693 ip-10-0-131-1 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:06:11.098693 ip-10-0-131-1 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:06:11.098693 ip-10-0-131-1 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:06:11.099305 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.098708 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:06:11.103244 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103230 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:06:11.103244 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103243 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:06:11.103244 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103246 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103250 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103253 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103256 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103260 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103262 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103265 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103268 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103270 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103273 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103276 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103278 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103286 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103289 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103292 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103294 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103297 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103299 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103302 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103304 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:06:11.103332 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103306 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103309 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103311 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103314 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103316 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103319 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103321 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103323 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103326 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103328 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103331 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103333 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103336 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103338 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103341 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103344 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103346 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103349 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103352 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:06:11.103786 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103356 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103366 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103368 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103371 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103373 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103376 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103378 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103380 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103383 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103385 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103387 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103390 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103392 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103395 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103397 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103401 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103403 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103406 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103410 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:06:11.104259 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103413 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103416 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103419 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103422 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103424 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103427 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103430 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103432 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103434 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103437 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103439 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103442 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103444 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103447 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103450 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103458 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103460 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103463 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103465 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103468 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103470 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:06:11.104705 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103473 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103476 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103478 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103482 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103485 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103488 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103491 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103494 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103497 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103500 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103503 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103505 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103508 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103510 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103513 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103515 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103518 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.103520 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104265 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:06:11.105206 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104271 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104274 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104277 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104280 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104283 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104286 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104288 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104291 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104295 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104297 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104300 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104303 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104305 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104307 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104310 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104313 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104315 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104317 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104320 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104322 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:06:11.105652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104325 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104328 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104330 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104333 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104335 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104338 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104340 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104343 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104345 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104349 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104353 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104357 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104360 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104363 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104367 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104369 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104372 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104374 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104376 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:06:11.106171 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104379 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104381 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104384 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104386 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104388 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104391 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104393 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104395 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104398 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104400 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104402 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104405 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104407 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104410 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104412 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104415 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104418 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104420 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104423 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104426 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:06:11.106639 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104428 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104431 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104434 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104436 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104439 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104441 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104444 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104448 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104451 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104453 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104455 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104458 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104460 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104462 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104465 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104468 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104472 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104476 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104479 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:06:11.107132 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104481 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104484 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104486 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104489 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104491 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104493 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104496 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104499 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104501 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104504 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104506 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104508 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104511 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104513 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104516 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104518 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104521 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104523 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104525 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104528 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:06:11.107558 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104889 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104894 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104897 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104900 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104902 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104905 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104907 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104910 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104913 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104916 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104918 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104920 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104923 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104926 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104928 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104930 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104933 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104935 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104938 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104941 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:06:11.108038 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104943 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104946 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104948 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104951 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104953 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104956 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104959 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104961 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104963 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104966 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104968 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104971 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104975 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104977 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104980 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104982 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104985 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104987 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104989 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104992 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:06:11.108480 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104994 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104997 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.104999 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105002 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105005 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105007 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105009 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105012 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105014 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105019 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105024 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105027 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105030 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105034 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105037 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105039 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105042 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105044 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105047 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:06:11.108960 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105050 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105052 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105057 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105060 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105063 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105067 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105070 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105073 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105075 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105078 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105080 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105083 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105085 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105087 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105090 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105093 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105095 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105098 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105100 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:06:11.109422 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105102 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105105 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105107 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105109 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105113 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105116 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105118 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105121 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105123 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105126 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105128 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105131 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105133 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105135 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105138 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105140 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105143 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105145 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105149 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105151 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105154 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:06:11.109899 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105221 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105227 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105232 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105236 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105240 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105243 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105247 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105251 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105254 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105257 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105260 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105264 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105266 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105269 2569 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105272 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105275 2569 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105278 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105280 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105285 2569 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105288 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105291 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105294 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105297 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105300 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:06:11.110386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105303 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105306 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105309 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105312 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105314 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105319 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105321 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105324 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105328 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105331 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105334 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105336 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105339 2569 flags.go:64] FLAG: --enable-server="true" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105341 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105345 2569 flags.go:64] FLAG: --event-burst="100" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105348 2569 flags.go:64] FLAG: --event-qps="50" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105350 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105354 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105356 2569 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105360 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105362 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105365 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105368 2569 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105371 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105373 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:06:11.110939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105376 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105379 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105382 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105385 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105388 2569 flags.go:64] FLAG: --feature-gates="" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105391 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105394 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105396 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105399 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105402 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105405 2569 flags.go:64] FLAG: --help="false" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105407 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-131-1.ec2.internal" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105412 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105415 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105417 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105421 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105424 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105427 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105429 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105432 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105435 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105438 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105440 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105443 2569 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:06:11.111576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105446 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105449 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105452 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105455 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105457 2569 flags.go:64] FLAG: --lock-file="" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105460 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105463 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105466 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105471 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105474 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105477 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105479 2569 flags.go:64] FLAG: --logging-format="text" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105482 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105485 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105488 2569 flags.go:64] FLAG: --manifest-url="" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105491 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105494 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105497 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105501 2569 flags.go:64] FLAG: --max-pods="110" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105504 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105508 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105511 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105513 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105516 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105519 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:06:11.112186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105522 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105528 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105531 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105534 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105536 2569 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105539 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105544 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105547 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105550 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105553 2569 flags.go:64] FLAG: --port="10250" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105556 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105559 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04e13bf6238d84c98" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105561 2569 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105564 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105567 2569 flags.go:64] FLAG: --register-node="true" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105570 2569 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105573 2569 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105576 2569 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105578 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105581 2569 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105584 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105587 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105590 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105592 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105595 2569 flags.go:64] FLAG: --runonce="false" Jul 06 19:06:11.112770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105597 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105600 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105604 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105607 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105609 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105612 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105615 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105618 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105620 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105623 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105625 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105628 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105631 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105633 2569 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105636 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105641 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105643 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105646 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105651 2569 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105654 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105656 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105660 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105663 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105666 2569 flags.go:64] FLAG: --v="2" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105670 2569 flags.go:64] FLAG: --version="false" Jul 06 19:06:11.113346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105674 2569 flags.go:64] FLAG: --vmodule="" Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105677 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.105680 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105817 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105823 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105826 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105829 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105832 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105834 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105838 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105841 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105843 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105846 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105848 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105851 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105854 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105856 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105859 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105861 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105864 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:06:11.113952 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105866 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105869 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105871 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105874 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105876 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105879 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105882 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105884 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105895 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105898 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105901 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105903 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105906 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105909 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105911 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105914 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105916 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105919 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105922 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:06:11.114421 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105926 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105929 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105932 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105935 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105939 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105942 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105945 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105948 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105951 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105954 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105956 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105959 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105961 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105964 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105967 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105969 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105971 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105974 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105976 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:06:11.114883 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105980 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105982 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105986 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105989 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105992 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105996 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.105998 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106001 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106004 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106006 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106009 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106011 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106013 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106016 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106018 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106021 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106023 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106026 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106029 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106031 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:06:11.115356 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106033 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106036 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106038 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106041 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106043 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106046 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106048 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106050 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106053 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106055 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106058 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106061 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106063 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106069 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106073 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106075 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106078 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106080 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106083 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106085 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:06:11.115895 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106087 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106090 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106092 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.106095 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.106651 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.112790 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.112801 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.112834 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.112840 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112890 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112895 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112899 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112902 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112905 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112908 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:06:11.116352 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112910 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112913 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112916 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112918 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112921 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112923 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112926 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112929 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112932 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112934 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112937 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112940 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112943 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112945 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112950 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112953 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112956 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112959 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112961 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112964 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:06:11.116712 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112966 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112969 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112971 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112974 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112976 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112978 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112981 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112984 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112986 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112988 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112991 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112993 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112996 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.112999 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113002 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113006 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113008 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113011 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113013 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113016 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:06:11.117186 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113019 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113022 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113024 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113027 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113030 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113033 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113035 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113038 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113040 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113043 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113045 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113048 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113050 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113052 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113054 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113057 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113060 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113062 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113064 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113067 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:06:11.117644 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113069 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113072 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113074 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113076 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113079 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113081 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113084 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113086 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113088 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113091 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113094 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113096 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113099 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113101 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113104 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113106 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113109 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113111 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113114 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113116 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:06:11.118134 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113118 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113121 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113123 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113125 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113127 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113130 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113132 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113135 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113137 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113139 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113142 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113144 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113147 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.113151 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113282 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:06:11.118586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113286 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113288 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113291 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113293 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113296 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113298 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113301 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113303 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113305 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113308 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113311 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113313 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113315 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113318 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113326 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113329 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113331 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113334 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113336 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:06:11.118962 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113339 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113341 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113343 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113346 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113348 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113350 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113353 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113355 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113359 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113363 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113365 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113368 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113370 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113373 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113375 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113377 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113380 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113382 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113385 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:06:11.119402 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113387 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113390 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113392 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113394 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113397 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113399 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113401 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113403 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113406 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113413 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113416 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113419 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113421 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113423 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113427 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113430 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113433 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113436 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113438 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:06:11.119846 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113441 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113444 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113446 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113448 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113450 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113453 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113455 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113457 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113460 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113462 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113465 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113467 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113470 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113472 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113476 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113478 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113481 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113484 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113488 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113490 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:06:11.120278 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113492 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113496 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113499 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113506 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113509 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113512 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113514 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113516 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113519 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113522 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113524 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113526 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113529 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113532 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113534 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113537 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113539 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113542 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113544 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113546 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:06:11.120897 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:11.113549 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:06:11.121666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.113554 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:06:11.121666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.113645 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:06:11.121666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.115531 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:06:11.121666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.116917 2569 server.go:1019] "Starting client certificate rotation" Jul 06 19:06:11.121666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.117026 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:06:11.121666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.117068 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:06:11.121666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.120908 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:06:11.122878 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.122837 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:06:11.133846 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.133822 2569 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:06:11.137046 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.137030 2569 log.go:25] "Validated CRI v1 image API" Jul 06 19:06:11.137846 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.137819 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:06:11.140647 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.140629 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b7170002-d8e9-46d7-8b56-e4b35beeaaf3:/dev/nvme0n1p4 ee6225ee-bc69-4a47-b3df-c857988bab7b:/dev/nvme0n1p3] Jul 06 19:06:11.140709 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.140646 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:06:11.146710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.146603 2569 manager.go:216] Machine: {Timestamp:2026-07-06 19:06:11.145328723 +0000 UTC m=+0.337667463 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199104 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29e812cabb8b99d71360d82924e4c7 SystemUUID:ec29e812-cabb-8b99-d713-60d82924e4c7 BootID:1df26064-1a1a-4cc1-9149-f5b3742fa5fa Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:93:71:dc:f1:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:93:71:dc:f1:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:e1:66:a0:19:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:06:11.146710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.146706 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:06:11.146829 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.146779 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:06:11.147978 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.147954 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:06:11.148100 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.147980 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-1.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:06:11.149166 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.149156 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:06:11.149197 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.149169 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:06:11.149197 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.149176 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:06:11.149867 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.149851 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:06:11.149923 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.149856 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:06:11.151033 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.151023 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:06:11.153815 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.153794 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:06:11.153955 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.153945 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:06:11.153989 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.153974 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:06:11.154029 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.153989 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:06:11.154029 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.154007 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:06:11.154086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.154035 2569 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:06:11.154086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.154045 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:06:11.154397 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.154386 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:06:11.154442 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.154403 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:06:11.156771 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.156757 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:06:11.159429 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.159406 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:06:11.159505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.159455 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:06:11.160804 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160786 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:06:11.160881 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160842 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:06:11.160912 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160895 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:06:11.160912 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160904 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:06:11.160990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160914 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:06:11.160990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160920 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:06:11.160990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160928 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:06:11.160990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160933 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:06:11.160990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160941 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:06:11.160990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160946 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:06:11.160990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160956 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:06:11.160990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.160964 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:06:11.162062 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.162046 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:06:11.162127 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.162064 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:06:11.164384 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.164358 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:06:11.164384 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.164371 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-1.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:06:11.164384 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.164387 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-1.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:06:11.165122 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.165110 2569 server.go:1268] "Started kubelet" Jul 06 19:06:11.165217 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.165175 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:06:11.165282 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.165221 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:06:11.165336 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.165305 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:06:11.165911 ip-10-0-131-1 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:06:11.167167 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.167147 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:06:11.167509 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.167497 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:06:11.169562 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.169543 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:06:11.170975 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.170027 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-1.ec2.internal.18bfc8cf8a3d94d8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-1.ec2.internal,UID:ip-10-0-131-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-1.ec2.internal,},FirstTimestamp:2026-07-06 19:06:11.16508284 +0000 UTC m=+0.357421579,LastTimestamp:2026-07-06 19:06:11.16508284 +0000 UTC m=+0.357421579,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-1.ec2.internal,}" Jul 06 19:06:11.172641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.172622 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:06:11.172641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.172633 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:06:11.172797 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.172762 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:11.173129 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173116 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:06:11.173244 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173225 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:06:11.173335 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173119 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:06:11.173461 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173423 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:06:11.173540 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173463 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:06:11.173594 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173545 2569 factory.go:55] Registering systemd factory Jul 06 19:06:11.173594 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173558 2569 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:06:11.173751 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173737 2569 factory.go:153] Registering CRI-O factory Jul 06 19:06:11.173751 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173751 2569 factory.go:223] Registration of the crio container factory successfully Jul 06 19:06:11.173908 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173836 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:06:11.173908 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173858 2569 factory.go:103] Registering Raw factory Jul 06 19:06:11.173908 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173878 2569 manager.go:1196] Started watching for new ooms in manager Jul 06 19:06:11.174104 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.173936 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-csjvl" Jul 06 19:06:11.174253 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.174242 2569 manager.go:349] Starting recovery of all containers Jul 06 19:06:11.175274 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.175247 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:06:11.175474 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.175452 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-1.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:06:11.182391 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.182368 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-csjvl" Jul 06 19:06:11.186048 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.186030 2569 manager.go:354] Recovery completed Jul 06 19:06:11.187654 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.187636 2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 19:06:11.190483 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.190471 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:06:11.192170 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.192155 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:06:11.192237 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.192188 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:06:11.192237 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.192197 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:06:11.192586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.192573 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:06:11.192586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.192588 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:06:11.192586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.192593 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:06:11.192688 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.192605 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:06:11.193477 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.193460 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:06:11.193551 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.193485 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:06:11.193551 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.193494 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:06:11.193801 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.193791 2569 policy_none.go:49] "None policy: Start" Jul 06 19:06:11.193877 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.193821 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:06:11.193877 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.193833 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:06:11.194164 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.194106 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-1.ec2.internal.18bfc8cf8bdae941 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-1.ec2.internal,UID:ip-10-0-131-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-1.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-1.ec2.internal,},FirstTimestamp:2026-07-06 19:06:11.192170817 +0000 UTC m=+0.384509559,LastTimestamp:2026-07-06 19:06:11.192170817 +0000 UTC m=+0.384509559,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-1.ec2.internal,}" Jul 06 19:06:11.194986 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.194975 2569 policy_none.go:47] "Start" Jul 06 19:06:11.236418 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.236310 2569 manager.go:337] "Starting Device Plugin manager" Jul 06 19:06:11.236494 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.236436 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:06:11.236494 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.236449 2569 server.go:87] "Starting device plugin registration server" Jul 06 19:06:11.236665 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.236654 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:06:11.236693 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.236669 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:06:11.236803 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.236777 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:06:11.236908 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.236882 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:06:11.236908 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.236901 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:06:11.237292 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.237271 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:06:11.237371 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.237322 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:11.337280 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.337246 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:06:11.338646 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.338277 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:06:11.338646 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.338308 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:06:11.338646 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.338318 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:06:11.338646 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.338343 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-1.ec2.internal" Jul 06 19:06:11.346422 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.346408 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-1.ec2.internal" Jul 06 19:06:11.346480 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.346430 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-1.ec2.internal\": node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:11.359262 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.359213 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:11.362129 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.362101 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:06:11.363316 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.363301 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:06:11.363410 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.363331 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:06:11.363410 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.363350 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:06:11.363410 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.363380 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:06:11.365695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.365675 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:06:11.460004 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:11.459985 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:11.464137 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.464121 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal"] Jul 06 19:06:11.464206 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.464181 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:06:11.465454 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.465441 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:06:11.465516 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.465470 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:06:11.465516 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.465483 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:06:11.466959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.466946 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:06:11.467060 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.467040 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" Jul 06 19:06:11.467100 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.467065 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:06:11.467592 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.467576 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:06:11.467664 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.467604 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:06:11.467664 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.467615 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:06:11.467664 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.467582 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:06:11.467664 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.467649 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:06:11.467664 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.467663 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:06:11.468776 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.468761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:11.468857 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.468791 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:06:11.469453 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.469439 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:06:11.469515 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.469462 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:06:11.469515 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.469472 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:06:11.475424 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.475403 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1eb58fae0b7cc9453d7a3dd0280f4dd9-config\") pod \"kube-apiserver-proxy-ip-10-0-131-1.ec2.internal\" (UID: \"1eb58fae0b7cc9453d7a3dd0280f4dd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" Jul 06 19:06:11.475497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.475430 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e824a8f766307659a9006ac2592772e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal\" (UID: \"5e824a8f766307659a9006ac2592772e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:11.475497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:11.475447 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e824a8f766307659a9006ac2592772e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal\" (UID: \"5e824a8f766307659a9006ac2592772e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.144285 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.144271 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-1.ec2.internal\" not found" node="ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.148337 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.148323 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-1.ec2.internal\" not found" node="ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.212009 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.211989 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:13.227419 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.227400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e824a8f766307659a9006ac2592772e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal\" (UID: \"5e824a8f766307659a9006ac2592772e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.227565 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.227441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e824a8f766307659a9006ac2592772e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal\" (UID: \"5e824a8f766307659a9006ac2592772e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.227565 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.227477 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1eb58fae0b7cc9453d7a3dd0280f4dd9-config\") pod \"kube-apiserver-proxy-ip-10-0-131-1.ec2.internal\" (UID: \"1eb58fae0b7cc9453d7a3dd0280f4dd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.227565 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.227488 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e824a8f766307659a9006ac2592772e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal\" (UID: \"5e824a8f766307659a9006ac2592772e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.227565 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.227505 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e824a8f766307659a9006ac2592772e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal\" (UID: \"5e824a8f766307659a9006ac2592772e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.227565 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.227521 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1eb58fae0b7cc9453d7a3dd0280f4dd9-config\") pod \"kube-apiserver-proxy-ip-10-0-131-1.ec2.internal\" (UID: \"1eb58fae0b7cc9453d7a3dd0280f4dd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.312849 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.312799 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:13.413353 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.413334 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:13.447873 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.447847 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.451340 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.451321 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.514384 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.514361 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:13.614981 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.614918 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:13.664589 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.664571 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:06:13.715639 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.715621 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:13.768205 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.768182 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:06:13.768565 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.768306 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:06:13.768565 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.768342 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:06:13.816610 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:13.816590 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-1.ec2.internal\" not found" Jul 06 19:06:13.838099 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.838076 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:01:12 +0000 UTC" deadline="2027-12-14 22:31:32.290027675 +0000 UTC" Jul 06 19:06:13.838162 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.838100 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12627h25m18.451930542s" Jul 06 19:06:13.867309 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.867266 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:06:13.925142 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.925125 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.934255 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.934239 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:06:13.936343 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.936332 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" Jul 06 19:06:13.946293 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:13.946279 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:06:14.184483 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.183705 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:06:14.532697 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:14.532659 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb58fae0b7cc9453d7a3dd0280f4dd9.slice/crio-d7a07feafcff6d7f24edb95be95f6f335d93daa9d6a854a8163099d80f80ddd8 WatchSource:0}: Error finding container d7a07feafcff6d7f24edb95be95f6f335d93daa9d6a854a8163099d80f80ddd8: Status 404 returned error can't find the container with id d7a07feafcff6d7f24edb95be95f6f335d93daa9d6a854a8163099d80f80ddd8 Jul 06 19:06:14.533173 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:14.533151 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e824a8f766307659a9006ac2592772e.slice/crio-cd9ce0b3e7a9f9d2939cd026853e747d406aa53e22a981913fb294757e9871f5 WatchSource:0}: Error finding container cd9ce0b3e7a9f9d2939cd026853e747d406aa53e22a981913fb294757e9871f5: Status 404 returned error can't find the container with id cd9ce0b3e7a9f9d2939cd026853e747d406aa53e22a981913fb294757e9871f5 Jul 06 19:06:14.537313 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.537298 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:06:14.807325 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.807262 2569 apiserver.go:52] "Watching apiserver" Jul 06 19:06:14.813618 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.813599 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:06:14.813972 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.813949 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-d9c9k","openshift-image-registry/node-ca-v9t48","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal","openshift-multus/network-metrics-daemon-cwsgf","kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal","openshift-multus/multus-5czzp","openshift-multus/multus-additional-cni-plugins-rgpjd","openshift-network-diagnostics/network-check-target-tfvld","openshift-network-operator/iptables-alerter-n2k7w","openshift-ovn-kubernetes/ovnkube-node-n2lch","kube-system/global-pull-secret-syncer-fmm9q","kube-system/konnectivity-agent-cf7h9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr","openshift-cluster-node-tuning-operator/tuned-w8xdd"] Jul 06 19:06:14.816064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.816052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.817171 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.817156 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.818192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.818163 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dvdjr\"" Jul 06 19:06:14.818307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.818254 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:06:14.818392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.818316 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:06:14.818392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.818345 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:14.818524 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.818437 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:14.819546 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.819527 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:06:14.819546 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.819538 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:06:14.819715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.819531 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2stg7\"" Jul 06 19:06:14.819715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.819661 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:06:14.819825 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.819759 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.821104 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.821084 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.821735 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.821705 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qbwwn\"" Jul 06 19:06:14.821832 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.821758 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:06:14.821832 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.821770 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:06:14.822032 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.822020 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:06:14.822112 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.822023 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:06:14.822668 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.822652 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:14.822757 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.822735 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:14.823008 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.822993 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kq4vc\"" Jul 06 19:06:14.823216 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.823201 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:06:14.823294 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.823235 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:06:14.823891 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.823877 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.825301 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.825282 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.825372 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.825337 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:06:14.826547 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.826387 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fkm4f\"" Jul 06 19:06:14.826547 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.826455 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:06:14.826681 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.826547 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:06:14.826853 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.826820 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:06:14.827438 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.827293 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:14.827438 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.827397 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:14.828064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.828046 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:06:14.829174 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.828988 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:06:14.829174 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.829055 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s7cmv\"" Jul 06 19:06:14.830209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.829467 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:06:14.830209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.829693 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:06:14.830209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.830027 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:14.830209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.830034 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:06:14.831217 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.830780 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:06:14.832542 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.832498 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:14.832788 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.832618 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6kdmq\"" Jul 06 19:06:14.832788 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.832665 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:06:14.832788 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.832621 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:06:14.834383 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.834368 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.834507 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.834409 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:06:14.834910 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.834893 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qg8f\"" Jul 06 19:06:14.835004 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.834989 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:06:14.835209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.835196 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:06:14.836443 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.836404 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jqxjf\"" Jul 06 19:06:14.836515 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.836479 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:06:14.836885 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.836870 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:06:14.838112 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838097 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-socket-dir-parent\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838117 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-netns\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838131 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-hostroot\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-kubelet\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.838175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838159 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-systemd\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.838290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.838290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838216 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-run-netns\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.838290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-daemon-config\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838256 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-etc-kubernetes\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838271 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwmm\" (UniqueName: \"kubernetes.io/projected/92653146-3a2b-4388-a496-604a69771244-kube-api-access-phwmm\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.838444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838297 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ec7281-57ce-453e-80be-77e12bca6d52-tmp-dir\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.838444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838326 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f669448-6e2c-46f1-9e91-8ff68e9886a0-host\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.838444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-system-cni-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838363 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-conf-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838376 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-os-release\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.838444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838394 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-multus-certs\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f669448-6e2c-46f1-9e91-8ff68e9886a0-serviceca\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838459 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-cnibin\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838483 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovn-node-metrics-cert\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838523 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-os-release\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92653146-3a2b-4388-a496-604a69771244-host-slash\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838592 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-systemd-units\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838617 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17ec7281-57ce-453e-80be-77e12bca6d52-hosts-file\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838640 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-system-cni-dir\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-k8s-cni-cncf-io\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838687 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-var-lib-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.838730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838711 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-cni-bin\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovnkube-script-lib\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqzx\" (UniqueName: \"kubernetes.io/projected/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-kube-api-access-6jqzx\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk5d4\" (UniqueName: \"kubernetes.io/projected/17ec7281-57ce-453e-80be-77e12bca6d52-kube-api-access-rk5d4\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-cni-bin\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838904 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-node-log\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838927 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-log-socket\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838952 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-env-overrides\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.838975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2jp\" (UniqueName: \"kubernetes.io/projected/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-kube-api-access-dg2jp\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839010 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzfvx\" (UniqueName: \"kubernetes.io/projected/a8b29103-52ce-44a9-9d65-8c55bb83ed04-kube-api-access-mzfvx\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839097 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-kubelet\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.839192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839144 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-slash\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839159 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-etc-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4h4\" (UniqueName: \"kubernetes.io/projected/ed972654-4e81-41ad-b9ba-115c629b72fa-kube-api-access-bn4h4\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovnkube-config\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839218 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cnibin\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839228 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmrp\" (UniqueName: \"kubernetes.io/projected/5f669448-6e2c-46f1-9e91-8ff68e9886a0-kube-api-access-xvmrp\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-cni-multus\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839275 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839287 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-ovn\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-cni-netd\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed972654-4e81-41ad-b9ba-115c629b72fa-cni-binary-copy\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92653146-3a2b-4388-a496-604a69771244-iptables-alerter-script\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.839787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.839399 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-cni-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.864672 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.864659 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5967" Jul 06 19:06:14.872371 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.872353 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5967" Jul 06 19:06:14.926077 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.926062 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:06:14.940301 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-etc-kubernetes\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940388 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwmm\" (UniqueName: \"kubernetes.io/projected/92653146-3a2b-4388-a496-604a69771244-kube-api-access-phwmm\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.940388 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ec7281-57ce-453e-80be-77e12bca6d52-tmp-dir\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.940388 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f669448-6e2c-46f1-9e91-8ff68e9886a0-host\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.940388 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:14.940625 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940384 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-etc-kubernetes\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940625 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysconfig\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.940625 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940441 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f669448-6e2c-46f1-9e91-8ff68e9886a0-host\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.940625 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940477 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-system-cni-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940625 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-conf-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940625 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940549 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-system-cni-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940625 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-os-release\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.940625 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940615 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-conf-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-multus-certs\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940689 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17ec7281-57ce-453e-80be-77e12bca6d52-tmp-dir\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940696 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-os-release\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f669448-6e2c-46f1-9e91-8ff68e9886a0-serviceca\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940695 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-multus-certs\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-cnibin\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovn-node-metrics-cert\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940788 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e-agent-certs\") pod \"konnectivity-agent-cf7h9\" (UID: \"6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e\") " pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940802 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-os-release\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940819 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-cnibin\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92653146-3a2b-4388-a496-604a69771244-host-slash\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.940930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-systemd-units\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.940941 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92653146-3a2b-4388-a496-604a69771244-host-slash\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941002 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17ec7281-57ce-453e-80be-77e12bca6d52-hosts-file\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-system-cni-dir\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941032 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-os-release\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941059 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-sys-fs\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941033 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-systemd-units\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941090 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17ec7281-57ce-453e-80be-77e12bca6d52-hosts-file\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-host\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941115 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f669448-6e2c-46f1-9e91-8ff68e9886a0-serviceca\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941113 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941130 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-k8s-cni-cncf-io\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941167 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-var-lib-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941190 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-cni-bin\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941169 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-k8s-cni-cncf-io\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovnkube-script-lib\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941210 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-system-cni-dir\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqzx\" (UniqueName: \"kubernetes.io/projected/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-kube-api-access-6jqzx\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.941497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941248 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-var-lib-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941255 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-cni-bin\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5d4\" (UniqueName: \"kubernetes.io/projected/17ec7281-57ce-453e-80be-77e12bca6d52-kube-api-access-rk5d4\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941398 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941482 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941498 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-cni-bin\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941554 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-node-log\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-log-socket\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-env-overrides\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941620 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-cni-bin\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941660 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2jp\" (UniqueName: \"kubernetes.io/projected/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-kube-api-access-dg2jp\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941669 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-log-socket\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942078 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-socket-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942117 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-kubernetes\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.942172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09a32004-05e2-4a14-9943-7b9b3229cf2f-kubelet-config\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysctl-conf\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-run\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzfvx\" (UniqueName: \"kubernetes.io/projected/a8b29103-52ce-44a9-9d65-8c55bb83ed04-kube-api-access-mzfvx\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-sys\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942393 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-lib-modules\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-var-lib-kubelet\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942470 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942494 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b76a06f-31c5-4e9e-8d81-759953f85dbe-tmp\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942579 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-env-overrides\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.941624 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-node-log\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942809 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.942959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbbq\" (UniqueName: \"kubernetes.io/projected/8b76a06f-31c5-4e9e-8d81-759953f85dbe-kube-api-access-fxbbq\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-kubelet\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.942970 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-slash\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-etc-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943035 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovnkube-script-lib\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943054 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-769rh\" (UniqueName: \"kubernetes.io/projected/3ba6c99f-da5d-47a4-905e-0481714968a5-kube-api-access-769rh\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943130 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-modprobe-d\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943133 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-slash\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943146 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-kubelet\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4h4\" (UniqueName: \"kubernetes.io/projected/ed972654-4e81-41ad-b9ba-115c629b72fa-kube-api-access-bn4h4\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943222 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-etc-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovnkube-config\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cnibin\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943330 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-systemd\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-tuned\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943440 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-cnibin\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.943695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmrp\" (UniqueName: \"kubernetes.io/projected/5f669448-6e2c-46f1-9e91-8ff68e9886a0-kube-api-access-xvmrp\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.944400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943530 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e-konnectivity-ca\") pod \"konnectivity-agent-cf7h9\" (UID: \"6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e\") " pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:14.944400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943580 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-cni-multus\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.944400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:14.944400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943748 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-var-lib-cni-multus\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.944400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943801 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-ovn\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.944400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-cni-netd\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.944400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943938 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovnkube-config\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.944400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.943973 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-cni-netd\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.944758 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944725 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-ovn\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.944811 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944779 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:14.944903 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944813 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed972654-4e81-41ad-b9ba-115c629b72fa-cni-binary-copy\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.944903 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944841 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92653146-3a2b-4388-a496-604a69771244-iptables-alerter-script\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.944903 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-device-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:14.944903 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-cni-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-socket-dir-parent\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944957 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-netns\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.944983 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09a32004-05e2-4a14-9943-7b9b3229cf2f-dbus\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:14.945084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945011 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-registration-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:14.945084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945039 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-hostroot\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-kubelet\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.945084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945069 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-ovn-node-metrics-cert\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.945392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945092 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-systemd\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.945392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.945392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysctl-d\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:14.945392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945168 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-openvswitch\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.945392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-host-run-netns\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945392 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.945268 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:14.945392 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.945330 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs podName:a8b29103-52ce-44a9-9d65-8c55bb83ed04 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:15.445311806 +0000 UTC m=+2.986034029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs") pod "network-metrics-daemon-cwsgf" (UID: "a8b29103-52ce-44a9-9d65-8c55bb83ed04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:14.945764 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945691 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-hostroot\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945825 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945782 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-run-systemd\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.945825 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945795 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed972654-4e81-41ad-b9ba-115c629b72fa-cni-binary-copy\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945928 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945913 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-socket-dir-parent\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945976 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945922 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-cni-dir\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.945976 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.945965 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-kubelet\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.946064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.946001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-run-netns\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.946064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.946038 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-daemon-config\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.946236 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.946207 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-host-run-netns\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.946336 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.946317 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92653146-3a2b-4388-a496-604a69771244-iptables-alerter-script\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.946952 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.946928 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed972654-4e81-41ad-b9ba-115c629b72fa-multus-daemon-config\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:14.948584 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.948562 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwmm\" (UniqueName: \"kubernetes.io/projected/92653146-3a2b-4388-a496-604a69771244-kube-api-access-phwmm\") pod \"iptables-alerter-n2k7w\" (UID: \"92653146-3a2b-4388-a496-604a69771244\") " pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:14.948783 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.948767 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5d4\" (UniqueName: \"kubernetes.io/projected/17ec7281-57ce-453e-80be-77e12bca6d52-kube-api-access-rk5d4\") pod \"node-resolver-d9c9k\" (UID: \"17ec7281-57ce-453e-80be-77e12bca6d52\") " pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:14.948921 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.948901 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqzx\" (UniqueName: \"kubernetes.io/projected/7c06366a-b8aa-4ebe-bda8-afcdbbc12f66-kube-api-access-6jqzx\") pod \"ovnkube-node-n2lch\" (UID: \"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:14.949214 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.949201 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:06:14.949254 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.949218 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:06:14.949254 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.949226 2569 projected.go:196] Error preparing data for projected volume kube-api-access-wvcgn for pod openshift-network-diagnostics/network-check-target-tfvld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:14.949317 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:14.949267 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn podName:d8c1f921-195b-46c7-8245-02e16d2679d7 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:15.449256011 +0000 UTC m=+2.989978222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wvcgn" (UniqueName: "kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn") pod "network-check-target-tfvld" (UID: "d8c1f921-195b-46c7-8245-02e16d2679d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:14.950251 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.950235 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzfvx\" (UniqueName: \"kubernetes.io/projected/a8b29103-52ce-44a9-9d65-8c55bb83ed04-kube-api-access-mzfvx\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:14.950518 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.950498 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2jp\" (UniqueName: \"kubernetes.io/projected/5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca-kube-api-access-dg2jp\") pod \"multus-additional-cni-plugins-rgpjd\" (UID: \"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca\") " pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:14.951371 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.951354 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmrp\" (UniqueName: \"kubernetes.io/projected/5f669448-6e2c-46f1-9e91-8ff68e9886a0-kube-api-access-xvmrp\") pod \"node-ca-v9t48\" (UID: \"5f669448-6e2c-46f1-9e91-8ff68e9886a0\") " pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:14.951681 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:14.951667 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4h4\" (UniqueName: \"kubernetes.io/projected/ed972654-4e81-41ad-b9ba-115c629b72fa-kube-api-access-bn4h4\") pod \"multus-5czzp\" (UID: \"ed972654-4e81-41ad-b9ba-115c629b72fa\") " pod="openshift-multus/multus-5czzp" Jul 06 19:06:15.018813 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.018777 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" event={"ID":"5e824a8f766307659a9006ac2592772e","Type":"ContainerStarted","Data":"cd9ce0b3e7a9f9d2939cd026853e747d406aa53e22a981913fb294757e9871f5"} Jul 06 19:06:15.019699 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.019682 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" event={"ID":"1eb58fae0b7cc9453d7a3dd0280f4dd9","Type":"ContainerStarted","Data":"d7a07feafcff6d7f24edb95be95f6f335d93daa9d6a854a8163099d80f80ddd8"} Jul 06 19:06:15.047013 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.046992 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09a32004-05e2-4a14-9943-7b9b3229cf2f-dbus\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:15.047135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047024 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-registration-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047050 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysctl-d\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047115 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-registration-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047115 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09a32004-05e2-4a14-9943-7b9b3229cf2f-dbus\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047138 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047187 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysconfig\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047212 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysctl-d\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e-agent-certs\") pod \"konnectivity-agent-cf7h9\" (UID: \"6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e\") " pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-sys-fs\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047269 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysconfig\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047269 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-host\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.047305 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047334 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-sys-fs\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.047350 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret podName:09a32004-05e2-4a14-9943-7b9b3229cf2f nodeName:}" failed. No retries permitted until 2026-07-06 19:06:15.547333703 +0000 UTC m=+3.088055921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret") pod "global-pull-secret-syncer-fmm9q" (UID: "09a32004-05e2-4a14-9943-7b9b3229cf2f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047366 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-socket-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047379 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047382 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-host\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-kubernetes\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047447 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-kubernetes\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047474 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-socket-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09a32004-05e2-4a14-9943-7b9b3229cf2f-kubelet-config\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047632 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysctl-conf\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-run\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.047739 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047682 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-sys\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047709 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09a32004-05e2-4a14-9943-7b9b3229cf2f-kubelet-config\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047802 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-sysctl-conf\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047736 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-lib-modules\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047850 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-run\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047803 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-sys\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-var-lib-kubelet\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b76a06f-31c5-4e9e-8d81-759953f85dbe-tmp\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047919 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-lib-modules\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047937 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbbq\" (UniqueName: \"kubernetes.io/projected/8b76a06f-31c5-4e9e-8d81-759953f85dbe-kube-api-access-fxbbq\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047979 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-var-lib-kubelet\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.047999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-769rh\" (UniqueName: \"kubernetes.io/projected/3ba6c99f-da5d-47a4-905e-0481714968a5-kube-api-access-769rh\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-modprobe-d\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-systemd\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-tuned\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048134 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e-konnectivity-ca\") pod \"konnectivity-agent-cf7h9\" (UID: \"6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e\") " pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048181 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-modprobe-d\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.048298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-device-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.049144 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048235 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ba6c99f-da5d-47a4-905e-0481714968a5-device-dir\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.049144 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-systemd\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.049144 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.048749 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e-konnectivity-ca\") pod \"konnectivity-agent-cf7h9\" (UID: \"6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e\") " pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:15.049606 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.049586 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e-agent-certs\") pod \"konnectivity-agent-cf7h9\" (UID: \"6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e\") " pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:15.050342 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.050321 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b76a06f-31c5-4e9e-8d81-759953f85dbe-etc-tuned\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.050715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.050700 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b76a06f-31c5-4e9e-8d81-759953f85dbe-tmp\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.056837 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.056818 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbbq\" (UniqueName: \"kubernetes.io/projected/8b76a06f-31c5-4e9e-8d81-759953f85dbe-kube-api-access-fxbbq\") pod \"tuned-w8xdd\" (UID: \"8b76a06f-31c5-4e9e-8d81-759953f85dbe\") " pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.056923 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.056845 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-769rh\" (UniqueName: \"kubernetes.io/projected/3ba6c99f-da5d-47a4-905e-0481714968a5-kube-api-access-769rh\") pod \"aws-ebs-csi-driver-node-fhtbr\" (UID: \"3ba6c99f-da5d-47a4-905e-0481714968a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.126798 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.126732 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d9c9k" Jul 06 19:06:15.132761 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.132742 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9t48" Jul 06 19:06:15.141077 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.141057 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5czzp" Jul 06 19:06:15.141771 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:15.141744 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ec7281_57ce_453e_80be_77e12bca6d52.slice/crio-700f3d9c0a4bffee6122967453073e1496b94622d2a4a0e050c195f5df18b31a WatchSource:0}: Error finding container 700f3d9c0a4bffee6122967453073e1496b94622d2a4a0e050c195f5df18b31a: Status 404 returned error can't find the container with id 700f3d9c0a4bffee6122967453073e1496b94622d2a4a0e050c195f5df18b31a Jul 06 19:06:15.144663 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.144618 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" Jul 06 19:06:15.145849 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:15.145825 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f669448_6e2c_46f1_9e91_8ff68e9886a0.slice/crio-575a42de002f4c5830924e2aca13bd9eaee1253fa620ac0725e9eaa0b5bffcde WatchSource:0}: Error finding container 575a42de002f4c5830924e2aca13bd9eaee1253fa620ac0725e9eaa0b5bffcde: Status 404 returned error can't find the container with id 575a42de002f4c5830924e2aca13bd9eaee1253fa620ac0725e9eaa0b5bffcde Jul 06 19:06:15.151107 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.151069 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n2k7w" Jul 06 19:06:15.153221 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:15.153195 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded972654_4e81_41ad_b9ba_115c629b72fa.slice/crio-6ad54c7a820ba1b7d42e92d46ca01f9e4c2889d82c546c676e8b9427cdda508e WatchSource:0}: Error finding container 6ad54c7a820ba1b7d42e92d46ca01f9e4c2889d82c546c676e8b9427cdda508e: Status 404 returned error can't find the container with id 6ad54c7a820ba1b7d42e92d46ca01f9e4c2889d82c546c676e8b9427cdda508e Jul 06 19:06:15.156549 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.156531 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:15.161995 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.161950 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:15.167630 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.167479 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" Jul 06 19:06:15.171417 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.171398 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" Jul 06 19:06:15.171850 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:15.171818 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c06366a_b8aa_4ebe_bda8_afcdbbc12f66.slice/crio-9e65689c2ad5bca8b5b910c5624017dc455f7be815c37cde2c747d91e242b3b9 WatchSource:0}: Error finding container 9e65689c2ad5bca8b5b910c5624017dc455f7be815c37cde2c747d91e242b3b9: Status 404 returned error can't find the container with id 9e65689c2ad5bca8b5b910c5624017dc455f7be815c37cde2c747d91e242b3b9 Jul 06 19:06:15.180859 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:15.180802 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad6fd62_c4cf_47c3_a1aa_4d71e1fe245e.slice/crio-3db388e47bada784a9a15ddf20daf509983b06d4e735c0a9ca4b83f4b481b284 WatchSource:0}: Error finding container 3db388e47bada784a9a15ddf20daf509983b06d4e735c0a9ca4b83f4b481b284: Status 404 returned error can't find the container with id 3db388e47bada784a9a15ddf20daf509983b06d4e735c0a9ca4b83f4b481b284 Jul 06 19:06:15.191095 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:15.191059 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b76a06f_31c5_4e9e_8d81_759953f85dbe.slice/crio-fa6693847f816eb6027eb0c745b4de63bdbdce18958ab847e2ea85a6c3d2a992 WatchSource:0}: Error finding container fa6693847f816eb6027eb0c745b4de63bdbdce18958ab847e2ea85a6c3d2a992: Status 404 returned error can't find the container with id fa6693847f816eb6027eb0c745b4de63bdbdce18958ab847e2ea85a6c3d2a992 Jul 06 19:06:15.221090 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.221069 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:06:15.450184 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.450110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:15.450184 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.450156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:15.450391 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.450296 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:06:15.450391 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.450322 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:06:15.450391 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.450335 2569 projected.go:196] Error preparing data for projected volume kube-api-access-wvcgn for pod openshift-network-diagnostics/network-check-target-tfvld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:15.450391 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.450303 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:15.450391 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.450389 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn podName:d8c1f921-195b-46c7-8245-02e16d2679d7 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:16.450372404 +0000 UTC m=+3.991094610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvcgn" (UniqueName: "kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn") pod "network-check-target-tfvld" (UID: "d8c1f921-195b-46c7-8245-02e16d2679d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:15.450688 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.450443 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs podName:a8b29103-52ce-44a9-9d65-8c55bb83ed04 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:16.450412593 +0000 UTC m=+3.991134816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs") pod "network-metrics-daemon-cwsgf" (UID: "a8b29103-52ce-44a9-9d65-8c55bb83ed04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:15.551360 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.551327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:15.551530 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.551511 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:15.551596 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:15.551579 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret podName:09a32004-05e2-4a14-9943-7b9b3229cf2f nodeName:}" failed. No retries permitted until 2026-07-06 19:06:16.55156058 +0000 UTC m=+4.092282791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret") pod "global-pull-secret-syncer-fmm9q" (UID: "09a32004-05e2-4a14-9943-7b9b3229cf2f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:15.874600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.874446 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:01:14 +0000 UTC" deadline="2028-01-27 21:30:45.139450853 +0000 UTC" Jul 06 19:06:15.874600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.874502 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13682h24m29.264952056s" Jul 06 19:06:15.928832 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:15.928801 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:06:16.015830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.015800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:16.016003 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.015928 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:16.016064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.016008 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:16.016110 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.016090 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:16.016161 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.016144 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:16.016227 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.016207 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:16.024988 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.024958 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" event={"ID":"8b76a06f-31c5-4e9e-8d81-759953f85dbe","Type":"ContainerStarted","Data":"fa6693847f816eb6027eb0c745b4de63bdbdce18958ab847e2ea85a6c3d2a992"} Jul 06 19:06:16.027355 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.027299 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" event={"ID":"3ba6c99f-da5d-47a4-905e-0481714968a5","Type":"ContainerStarted","Data":"06d454067f1133d7e4a86546dd1c0f19eddc75ee7ac0aaa4be088f57a642d50b"} Jul 06 19:06:16.030752 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.030725 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cf7h9" event={"ID":"6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e","Type":"ContainerStarted","Data":"3db388e47bada784a9a15ddf20daf509983b06d4e735c0a9ca4b83f4b481b284"} Jul 06 19:06:16.035562 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.035537 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"9e65689c2ad5bca8b5b910c5624017dc455f7be815c37cde2c747d91e242b3b9"} Jul 06 19:06:16.040172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.040147 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n2k7w" event={"ID":"92653146-3a2b-4388-a496-604a69771244","Type":"ContainerStarted","Data":"1e8f1c98bd326ef81ae23770ee9bf8e05e3e37c76f1913abfda394357510d849"} Jul 06 19:06:16.043881 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.043856 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9t48" event={"ID":"5f669448-6e2c-46f1-9e91-8ff68e9886a0","Type":"ContainerStarted","Data":"575a42de002f4c5830924e2aca13bd9eaee1253fa620ac0725e9eaa0b5bffcde"} Jul 06 19:06:16.049572 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.049481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerStarted","Data":"2f944b76285cdaeb78d4fe87ac26b5cad328c5b8acf892dab80c21d58536fa78"} Jul 06 19:06:16.054567 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.054544 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5czzp" event={"ID":"ed972654-4e81-41ad-b9ba-115c629b72fa","Type":"ContainerStarted","Data":"6ad54c7a820ba1b7d42e92d46ca01f9e4c2889d82c546c676e8b9427cdda508e"} Jul 06 19:06:16.057925 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.057903 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d9c9k" event={"ID":"17ec7281-57ce-453e-80be-77e12bca6d52","Type":"ContainerStarted","Data":"700f3d9c0a4bffee6122967453073e1496b94622d2a4a0e050c195f5df18b31a"} Jul 06 19:06:16.460062 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.460028 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:16.460237 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.460078 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:16.460301 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.460252 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:16.460349 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.460314 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs podName:a8b29103-52ce-44a9-9d65-8c55bb83ed04 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:18.460295689 +0000 UTC m=+6.001017899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs") pod "network-metrics-daemon-cwsgf" (UID: "a8b29103-52ce-44a9-9d65-8c55bb83ed04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:16.460850 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.460403 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:06:16.460850 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.460421 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:06:16.460850 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.460447 2569 projected.go:196] Error preparing data for projected volume kube-api-access-wvcgn for pod openshift-network-diagnostics/network-check-target-tfvld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:16.460850 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.460485 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn podName:d8c1f921-195b-46c7-8245-02e16d2679d7 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:18.460473472 +0000 UTC m=+6.001195682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvcgn" (UniqueName: "kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn") pod "network-check-target-tfvld" (UID: "d8c1f921-195b-46c7-8245-02e16d2679d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:16.561319 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.560751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:16.561319 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.560937 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:16.561319 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:16.560997 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret podName:09a32004-05e2-4a14-9943-7b9b3229cf2f nodeName:}" failed. No retries permitted until 2026-07-06 19:06:18.560979451 +0000 UTC m=+6.101701662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret") pod "global-pull-secret-syncer-fmm9q" (UID: "09a32004-05e2-4a14-9943-7b9b3229cf2f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:16.875011 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.874921 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:01:14 +0000 UTC" deadline="2028-02-29 09:27:07.138784615 +0000 UTC" Jul 06 19:06:16.875011 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:16.874963 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14462h20m50.263826028s" Jul 06 19:06:18.015858 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:18.015827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:18.015858 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:18.015866 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:18.016312 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:18.015827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:18.016312 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.015951 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:18.016312 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.016072 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:18.016312 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.016175 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:18.476841 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:18.476683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:18.476841 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:18.476729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:18.476841 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.476838 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:18.477107 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.476857 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:06:18.477107 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.476883 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:06:18.477107 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.476897 2569 projected.go:196] Error preparing data for projected volume kube-api-access-wvcgn for pod openshift-network-diagnostics/network-check-target-tfvld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:18.477107 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.476903 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs podName:a8b29103-52ce-44a9-9d65-8c55bb83ed04 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:22.476883157 +0000 UTC m=+10.017605372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs") pod "network-metrics-daemon-cwsgf" (UID: "a8b29103-52ce-44a9-9d65-8c55bb83ed04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:18.477107 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.476950 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn podName:d8c1f921-195b-46c7-8245-02e16d2679d7 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:22.476932774 +0000 UTC m=+10.017654986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvcgn" (UniqueName: "kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn") pod "network-check-target-tfvld" (UID: "d8c1f921-195b-46c7-8245-02e16d2679d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:18.577746 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:18.577707 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:18.577905 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.577848 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:18.577986 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:18.577911 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret podName:09a32004-05e2-4a14-9943-7b9b3229cf2f nodeName:}" failed. No retries permitted until 2026-07-06 19:06:22.577892714 +0000 UTC m=+10.118614940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret") pod "global-pull-secret-syncer-fmm9q" (UID: "09a32004-05e2-4a14-9943-7b9b3229cf2f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:20.016230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:20.016198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:20.016718 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:20.016331 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:20.016778 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:20.016748 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:20.016868 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:20.016838 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:20.016940 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:20.016903 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:20.016992 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:20.016975 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:22.015795 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:22.015763 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:22.015795 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:22.015773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:22.016171 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:22.015773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:22.016171 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.015852 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:22.016171 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.015917 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:22.016171 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.016022 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:22.508440 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:22.508123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:22.508440 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:22.508170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:22.508440 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.508261 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:06:22.508440 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.508285 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:06:22.508440 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.508297 2569 projected.go:196] Error preparing data for projected volume kube-api-access-wvcgn for pod openshift-network-diagnostics/network-check-target-tfvld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:22.508440 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.508298 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:22.508440 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.508350 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs podName:a8b29103-52ce-44a9-9d65-8c55bb83ed04 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:30.50833153 +0000 UTC m=+18.049053742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs") pod "network-metrics-daemon-cwsgf" (UID: "a8b29103-52ce-44a9-9d65-8c55bb83ed04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:22.508440 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.508367 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn podName:d8c1f921-195b-46c7-8245-02e16d2679d7 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:30.508359419 +0000 UTC m=+18.049081626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvcgn" (UniqueName: "kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn") pod "network-check-target-tfvld" (UID: "d8c1f921-195b-46c7-8245-02e16d2679d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:22.608484 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:22.608441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:22.608638 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.608555 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:22.608638 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:22.608634 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret podName:09a32004-05e2-4a14-9943-7b9b3229cf2f nodeName:}" failed. No retries permitted until 2026-07-06 19:06:30.608619808 +0000 UTC m=+18.149342034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret") pod "global-pull-secret-syncer-fmm9q" (UID: "09a32004-05e2-4a14-9943-7b9b3229cf2f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:24.016633 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:24.016577 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:24.017100 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:24.016581 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:24.017100 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:24.016580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:24.017100 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:24.016711 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:24.017100 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:24.016838 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:24.017100 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:24.016896 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:26.015778 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:26.015708 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:26.015778 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:26.015741 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:26.016303 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:26.015836 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:26.016303 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:26.015898 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:26.016303 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:26.016022 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:26.016303 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:26.016097 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:28.015897 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:28.015864 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:28.016278 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:28.015896 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:28.016278 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:28.015979 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:28.016278 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:28.016059 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:28.016278 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:28.016067 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:28.016278 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:28.016147 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:30.016003 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:30.015969 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:30.016514 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:30.016093 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:30.016514 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.016106 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:30.016514 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:30.016157 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:30.016514 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.016238 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:30.016514 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.016304 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:30.573413 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:30.573374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:30.573696 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:30.573436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:30.573696 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.573585 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:06:30.573696 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.573597 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:30.573696 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.573608 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:06:30.573696 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.573618 2569 projected.go:196] Error preparing data for projected volume kube-api-access-wvcgn for pod openshift-network-diagnostics/network-check-target-tfvld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:30.573696 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.573659 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs podName:a8b29103-52ce-44a9-9d65-8c55bb83ed04 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:46.573639582 +0000 UTC m=+34.114361800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs") pod "network-metrics-daemon-cwsgf" (UID: "a8b29103-52ce-44a9-9d65-8c55bb83ed04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:30.573696 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.573676 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn podName:d8c1f921-195b-46c7-8245-02e16d2679d7 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:46.573667565 +0000 UTC m=+34.114389774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvcgn" (UniqueName: "kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn") pod "network-check-target-tfvld" (UID: "d8c1f921-195b-46c7-8245-02e16d2679d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:30.674659 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:30.674623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:30.674829 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.674781 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:30.674895 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:30.674849 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret podName:09a32004-05e2-4a14-9943-7b9b3229cf2f nodeName:}" failed. No retries permitted until 2026-07-06 19:06:46.674830988 +0000 UTC m=+34.215553194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret") pod "global-pull-secret-syncer-fmm9q" (UID: "09a32004-05e2-4a14-9943-7b9b3229cf2f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:32.015755 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:32.015552 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:32.016164 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:32.015553 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:32.016164 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:32.015839 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:32.016164 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:32.015937 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:32.016164 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:32.015552 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:32.016164 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:32.016035 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:34.016008 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.015525 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:34.016008 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.015523 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:34.016008 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:34.015650 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:34.016008 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:34.015784 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:34.016008 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.015540 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:34.016008 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:34.015887 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:34.094469 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.094417 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" event={"ID":"1eb58fae0b7cc9453d7a3dd0280f4dd9","Type":"ContainerStarted","Data":"d667079340d158594ea09871b5e853570e393e9fe2534b532f658a46accad168"} Jul 06 19:06:34.096888 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.096857 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" event={"ID":"8b76a06f-31c5-4e9e-8d81-759953f85dbe","Type":"ContainerStarted","Data":"3491e30bf16b1fc0c80d53dd74a60c072042f35a43e7e6694461ec4405c31ca0"} Jul 06 19:06:34.103375 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.103357 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:06:34.103761 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.103743 2569 generic.go:356] "Generic (PLEG): container finished" podID="7c06366a-b8aa-4ebe-bda8-afcdbbc12f66" containerID="3b55d4144d49a1b0b32b2277dd8f30a71b224d2cea7168cc24b649c8c1953c9e" exitCode=1 Jul 06 19:06:34.103852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.103797 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"03db81d1ddc2539ac79da8c92ae147f09e4882f5b9532c3af580e7d9cbd58095"} Jul 06 19:06:34.103904 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.103868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"18e6024cbfbef0464b78f913d24bfacc6807a1af530bbdf0a2d938c0b992dd08"} Jul 06 19:06:34.103963 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.103917 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"9d17200071cff32c4b774bccd915faf537e74bc3a6a636077f04fd66223b8788"} Jul 06 19:06:34.104003 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.103964 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"b0ceb055da2a1982b8afb5e9ee0c7ec411086fc627251923bb39c2b4f59e0abc"} Jul 06 19:06:34.104031 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.104022 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerDied","Data":"3b55d4144d49a1b0b32b2277dd8f30a71b224d2cea7168cc24b649c8c1953c9e"} Jul 06 19:06:34.104084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.104070 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"3af40c86fffa0af5f4a205901bc93f9bae6cf949e0245b9f6543be027080e4c3"} Jul 06 19:06:34.105259 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.105237 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5czzp" event={"ID":"ed972654-4e81-41ad-b9ba-115c629b72fa","Type":"ContainerStarted","Data":"2ccd63ff0836e52b5437ac3753ff09832a9dfcc829e480e91ff9e647ad93a4ef"} Jul 06 19:06:34.108324 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.108289 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-1.ec2.internal" podStartSLOduration=21.10827882 podStartE2EDuration="21.10827882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:34.107958827 +0000 UTC m=+21.648681055" watchObservedRunningTime="2026-07-06 19:06:34.10827882 +0000 UTC m=+21.649001048" Jul 06 19:06:34.143752 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.142928 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w8xdd" podStartSLOduration=3.381152239 podStartE2EDuration="21.142911529s" totalImagesPullingTime="17.76175929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:34.142547719 +0000 UTC m=+21.683269959" watchObservedRunningTime="2026-07-06 19:06:34.142911529 +0000 UTC m=+21.683633758" Jul 06 19:06:34.143752 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:34.143246 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5czzp" podStartSLOduration=2.521623947 podStartE2EDuration="21.143234339s" totalImagesPullingTime="18.621610392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:34.124786787 +0000 UTC m=+21.665509015" watchObservedRunningTime="2026-07-06 19:06:34.143234339 +0000 UTC m=+21.683956568" Jul 06 19:06:35.108773 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.108740 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca" containerID="5b7a715b4eda8675c4b549f3b78946966b5ee8d4421edd98fd6bb507a12a0552" exitCode=0 Jul 06 19:06:35.109386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.108827 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerDied","Data":"5b7a715b4eda8675c4b549f3b78946966b5ee8d4421edd98fd6bb507a12a0552"} Jul 06 19:06:35.110334 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.110315 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d9c9k" event={"ID":"17ec7281-57ce-453e-80be-77e12bca6d52","Type":"ContainerStarted","Data":"cdc278689d7d892b28b77fa59702ccf716a540b2e10e3107e2d0c5e85836820b"} Jul 06 19:06:35.111762 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.111745 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" event={"ID":"3ba6c99f-da5d-47a4-905e-0481714968a5","Type":"ContainerStarted","Data":"38b3d4d3d8d69271f7711769e5b037e1b9cbad513254fb30ae3f9078f000b3e7"} Jul 06 19:06:35.113082 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.113063 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cf7h9" event={"ID":"6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e","Type":"ContainerStarted","Data":"4f71cd17cf394de79154d1ae964415a8734b5cbb73b589c0a902b6fe61e8a5f8"} Jul 06 19:06:35.114378 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.114356 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n2k7w" event={"ID":"92653146-3a2b-4388-a496-604a69771244","Type":"ContainerStarted","Data":"5fa3d70c5edd3c9f4ae8637d097dd6e609538a68346a283b490f9bec65ea1221"} Jul 06 19:06:35.115675 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.115653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9t48" event={"ID":"5f669448-6e2c-46f1-9e91-8ff68e9886a0","Type":"ContainerStarted","Data":"f68610a91882b28fe9dbf85d3f620538250b19cfc6b93107034297bce7388a29"} Jul 06 19:06:35.116932 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.116915 2569 generic.go:356] "Generic (PLEG): container finished" podID="5e824a8f766307659a9006ac2592772e" containerID="4b88cac1a7376474316d853bbe670fdc2333d249a6c93b2fdb07d4294bb0cfeb" exitCode=0 Jul 06 19:06:35.117015 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.116974 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" event={"ID":"5e824a8f766307659a9006ac2592772e","Type":"ContainerDied","Data":"4b88cac1a7376474316d853bbe670fdc2333d249a6c93b2fdb07d4294bb0cfeb"} Jul 06 19:06:35.156999 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.156822 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d9c9k" podStartSLOduration=4.346026709 podStartE2EDuration="22.156809699s" totalImagesPullingTime="17.81078299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:35.156777049 +0000 UTC m=+22.697499277" watchObservedRunningTime="2026-07-06 19:06:35.156809699 +0000 UTC m=+22.697531925" Jul 06 19:06:35.183645 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.183608 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v9t48" podStartSLOduration=4.375720528 podStartE2EDuration="22.183593326s" totalImagesPullingTime="17.807872798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:35.170552826 +0000 UTC m=+22.711275056" watchObservedRunningTime="2026-07-06 19:06:35.183593326 +0000 UTC m=+22.724315557" Jul 06 19:06:35.183944 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.183919 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n2k7w" podStartSLOduration=8.434805479 podStartE2EDuration="22.183910568s" totalImagesPullingTime="13.749105089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:35.183542713 +0000 UTC m=+22.724264941" watchObservedRunningTime="2026-07-06 19:06:35.183910568 +0000 UTC m=+22.724632796" Jul 06 19:06:35.197453 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.197409 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cf7h9" podStartSLOduration=4.128542977 podStartE2EDuration="22.197399491s" totalImagesPullingTime="18.068856514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:35.197203214 +0000 UTC m=+22.737925442" watchObservedRunningTime="2026-07-06 19:06:35.197399491 +0000 UTC m=+22.738121717" Jul 06 19:06:35.495086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.495061 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:06:35.899480 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.899385 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:06:35.495083944Z","UUID":"4efa5cd5-27f1-4462-82bb-1bfb3d731559","Handler":null,"Name":"","Endpoint":""} Jul 06 19:06:35.900817 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.900796 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:06:35.900938 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:35.900838 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:06:36.015996 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.015963 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:36.016156 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:36.016103 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:36.016212 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.016158 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:36.016275 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.016258 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:36.016357 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:36.016252 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:36.016357 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:36.016346 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:36.120131 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.120102 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" event={"ID":"5e824a8f766307659a9006ac2592772e","Type":"ContainerStarted","Data":"7046ec579e5a22839972818bdcf36ce127b8fe5153cdfa6ce5118855cbafdca0"} Jul 06 19:06:36.122229 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.122198 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" event={"ID":"3ba6c99f-da5d-47a4-905e-0481714968a5","Type":"ContainerStarted","Data":"4faef883e45c3004317335cef7f326eaa2c0d720b9ffc6313a165933095f293f"} Jul 06 19:06:36.125768 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.125751 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:06:36.126180 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.126157 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"63ac4bd0cdb6c66498d4a666616a007548aa716e9af39d9b214cdd2d897154bc"} Jul 06 19:06:36.134543 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.134490 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-1.ec2.internal" podStartSLOduration=23.134476623 podStartE2EDuration="23.134476623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:36.134034955 +0000 UTC m=+23.674757184" watchObservedRunningTime="2026-07-06 19:06:36.134476623 +0000 UTC m=+23.675198852" Jul 06 19:06:36.846897 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.846865 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:36.847450 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:36.847418 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:37.132212 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:37.132137 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" event={"ID":"3ba6c99f-da5d-47a4-905e-0481714968a5","Type":"ContainerStarted","Data":"b89e0d3855cd808b228481144c30eda800e9a44e0bc701bd113edbaa95ff5768"} Jul 06 19:06:37.132676 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:37.132391 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:37.132782 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:37.132764 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cf7h9" Jul 06 19:06:37.148678 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:37.148641 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fhtbr" podStartSLOduration=11.987199235 podStartE2EDuration="24.148628208s" totalImagesPullingTime="12.161428973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:37.147403795 +0000 UTC m=+24.688126170" watchObservedRunningTime="2026-07-06 19:06:37.148628208 +0000 UTC m=+24.689350435" Jul 06 19:06:38.016144 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:38.016103 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:38.016144 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:38.016137 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:38.016346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:38.016140 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:38.016346 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:38.016229 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:38.016346 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:38.016305 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:38.016493 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:38.016372 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:39.140783 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.140617 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:06:39.141245 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.141213 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"aa3c69b32dadf45097960a997862b1700a291d28c7ca99806d03c7eef2d38a8a"} Jul 06 19:06:39.141650 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.141623 2569 scope.go:117] "RemoveContainer" containerID="3b55d4144d49a1b0b32b2277dd8f30a71b224d2cea7168cc24b649c8c1953c9e" Jul 06 19:06:39.141742 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.141628 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:39.331130 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.331091 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:39.421551 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.421469 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:39.553796 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.553719 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:39.723105 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.723033 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:39.829236 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:39.829204 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:40.015860 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.015834 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:40.016004 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.015834 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:40.016004 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:40.015951 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:40.016085 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:40.016051 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:40.016085 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.015838 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:40.016185 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:40.016162 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:40.350872 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.350847 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fmm9q"] Jul 06 19:06:40.351377 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.350988 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:40.351377 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:40.351062 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:40.354069 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.354049 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cwsgf"] Jul 06 19:06:40.354184 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.354151 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:40.354285 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:40.354261 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:40.354816 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.354793 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tfvld"] Jul 06 19:06:40.354907 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.354895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:40.355006 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:40.354980 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:40.852957 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.852741 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:40.953254 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:40.953206 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:41.147557 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.147455 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca" containerID="26505585640b918a52f9358e92d7c63208b26d53dfa000292fce25b8b0a019b5" exitCode=0 Jul 06 19:06:41.147701 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.147573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerDied","Data":"26505585640b918a52f9358e92d7c63208b26d53dfa000292fce25b8b0a019b5"} Jul 06 19:06:41.153351 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.153331 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:06:41.153699 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.153678 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" event={"ID":"7c06366a-b8aa-4ebe-bda8-afcdbbc12f66","Type":"ContainerStarted","Data":"3f7c2810b8176322ceb7c667df09ce9aaae780c3c28d1e354d4e292ab2a80fbd"} Jul 06 19:06:41.153897 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.153876 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:41.153991 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.153951 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:41.168495 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.168415 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:41.168586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.168536 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:06:41.196054 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:41.196016 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" podStartSLOduration=10.059429374 podStartE2EDuration="28.196002944s" totalImagesPullingTime="18.13657357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:41.194524419 +0000 UTC m=+28.735246646" watchObservedRunningTime="2026-07-06 19:06:41.196002944 +0000 UTC m=+28.736725171" Jul 06 19:06:42.015752 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:42.015725 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:42.015752 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:42.015750 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:42.016271 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:42.015725 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:42.016271 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:42.015824 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:42.016271 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:42.015900 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:42.016271 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:42.015958 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:42.466553 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:42.466530 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:42.523578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:42.523549 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:43.160036 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:43.159998 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca" containerID="0cf9019992cd3efd12aac847cb01f032163c10aa1a977778386063d2e013e6ba" exitCode=0 Jul 06 19:06:43.160364 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:43.160083 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerDied","Data":"0cf9019992cd3efd12aac847cb01f032163c10aa1a977778386063d2e013e6ba"} Jul 06 19:06:43.179842 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:43.179808 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" podUID="7c06366a-b8aa-4ebe-bda8-afcdbbc12f66" containerName="ovnkube-controller" probeResult="failure" output="" Jul 06 19:06:44.015670 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:44.015641 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:44.015839 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:44.015649 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:44.015839 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:44.015751 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:44.015839 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:44.015773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:44.015987 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:44.015841 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:44.015987 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:44.015893 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:44.574035 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:44.574007 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:44.648896 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:44.648836 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:44.711850 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:44.711816 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:45.165532 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:45.165505 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca" containerID="96a34698774b63cf5cbafe78f0b61ff7b4f394e180b21a8a128f16e57ad61e41" exitCode=0 Jul 06 19:06:45.165685 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:45.165541 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerDied","Data":"96a34698774b63cf5cbafe78f0b61ff7b4f394e180b21a8a128f16e57ad61e41"} Jul 06 19:06:46.015850 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.015820 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:46.015850 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.015845 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:46.016310 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.015834 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:46.016310 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.015944 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cwsgf" podUID="a8b29103-52ce-44a9-9d65-8c55bb83ed04" Jul 06 19:06:46.016310 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.016032 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fmm9q" podUID="09a32004-05e2-4a14-9943-7b9b3229cf2f" Jul 06 19:06:46.016310 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.016109 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tfvld" podUID="d8c1f921-195b-46c7-8245-02e16d2679d7" Jul 06 19:06:46.599886 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.599857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:46.599886 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.599890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:46.600110 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.599991 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:46.600110 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.600011 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:06:46.600110 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.600040 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:06:46.600110 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.600051 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs podName:a8b29103-52ce-44a9-9d65-8c55bb83ed04 nodeName:}" failed. No retries permitted until 2026-07-06 19:07:18.6000361 +0000 UTC m=+66.140758306 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs") pod "network-metrics-daemon-cwsgf" (UID: "a8b29103-52ce-44a9-9d65-8c55bb83ed04") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:06:46.600110 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.600053 2569 projected.go:196] Error preparing data for projected volume kube-api-access-wvcgn for pod openshift-network-diagnostics/network-check-target-tfvld: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:46.600110 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.600100 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn podName:d8c1f921-195b-46c7-8245-02e16d2679d7 nodeName:}" failed. No retries permitted until 2026-07-06 19:07:18.600086301 +0000 UTC m=+66.140808507 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wvcgn" (UniqueName: "kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn") pod "network-check-target-tfvld" (UID: "d8c1f921-195b-46c7-8245-02e16d2679d7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:06:46.701242 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.701215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:46.701382 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.701317 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:46.701466 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:46.701385 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret podName:09a32004-05e2-4a14-9943-7b9b3229cf2f nodeName:}" failed. No retries permitted until 2026-07-06 19:07:18.701366408 +0000 UTC m=+66.242088617 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret") pod "global-pull-secret-syncer-fmm9q" (UID: "09a32004-05e2-4a14-9943-7b9b3229cf2f") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:06:46.752229 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.752207 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-1.ec2.internal" event="NodeReady" Jul 06 19:06:46.752359 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.752336 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:06:46.796244 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.796221 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kt26b"] Jul 06 19:06:46.815205 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.815180 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2vbxq"] Jul 06 19:06:46.815337 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.815315 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:46.818700 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.818678 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:06:46.818818 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.818715 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:06:46.819065 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.819043 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8npzr\"" Jul 06 19:06:46.839505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.839474 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kt26b"] Jul 06 19:06:46.839620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.839517 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2vbxq"] Jul 06 19:06:46.839620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.839535 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:46.839620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.839550 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-99ktz"] Jul 06 19:06:46.841872 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.841854 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-29q45\"" Jul 06 19:06:46.842178 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.842162 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:06:46.842266 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.842183 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:06:46.843303 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.843281 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:06:46.843575 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.843560 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:06:46.859409 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.859358 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-99ktz"] Jul 06 19:06:46.859529 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.859466 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-99ktz" Jul 06 19:06:46.862040 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.862025 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:06:46.862135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.862057 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:06:46.862381 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.862360 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:06:46.864794 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.864777 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ll82v\"" Jul 06 19:06:46.963876 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:46.963846 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:47.002688 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002664 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwgg8\" (UniqueName: \"kubernetes.io/projected/c4c8c169-88e1-4115-8387-7107d15cf4ca-kube-api-access-vwgg8\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.002817 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002702 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4c8c169-88e1-4115-8387-7107d15cf4ca-crio-socket\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.002817 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwnt\" (UniqueName: \"kubernetes.io/projected/10e66372-e151-4119-91da-bfdf10300f53-kube-api-access-wpwnt\") pod \"ingress-canary-99ktz\" (UID: \"10e66372-e151-4119-91da-bfdf10300f53\") " pod="openshift-ingress-canary/ingress-canary-99ktz" Jul 06 19:06:47.002939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36ecd44d-1d0b-499b-9a05-a017cb70fa44-config-volume\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.002939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4c8c169-88e1-4115-8387-7107d15cf4ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.002939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4c8c169-88e1-4115-8387-7107d15cf4ca-data-volume\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.002939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002911 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e66372-e151-4119-91da-bfdf10300f53-cert\") pod \"ingress-canary-99ktz\" (UID: \"10e66372-e151-4119-91da-bfdf10300f53\") " pod="openshift-ingress-canary/ingress-canary-99ktz" Jul 06 19:06:47.002939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002933 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ecd44d-1d0b-499b-9a05-a017cb70fa44-tmp-dir\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.003117 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002955 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4c8c169-88e1-4115-8387-7107d15cf4ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.003117 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ecd44d-1d0b-499b-9a05-a017cb70fa44-metrics-tls\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.003117 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.002996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wz5\" (UniqueName: \"kubernetes.io/projected/36ecd44d-1d0b-499b-9a05-a017cb70fa44-kube-api-access-p9wz5\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.064783 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.064759 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:47.103655 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgg8\" (UniqueName: \"kubernetes.io/projected/c4c8c169-88e1-4115-8387-7107d15cf4ca-kube-api-access-vwgg8\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.103782 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103661 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4c8c169-88e1-4115-8387-7107d15cf4ca-crio-socket\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.103782 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwnt\" (UniqueName: \"kubernetes.io/projected/10e66372-e151-4119-91da-bfdf10300f53-kube-api-access-wpwnt\") pod \"ingress-canary-99ktz\" (UID: \"10e66372-e151-4119-91da-bfdf10300f53\") " pod="openshift-ingress-canary/ingress-canary-99ktz" Jul 06 19:06:47.103887 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103786 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4c8c169-88e1-4115-8387-7107d15cf4ca-crio-socket\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.103887 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36ecd44d-1d0b-499b-9a05-a017cb70fa44-config-volume\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.103887 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103822 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4c8c169-88e1-4115-8387-7107d15cf4ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.103887 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4c8c169-88e1-4115-8387-7107d15cf4ca-data-volume\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.104063 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e66372-e151-4119-91da-bfdf10300f53-cert\") pod \"ingress-canary-99ktz\" (UID: \"10e66372-e151-4119-91da-bfdf10300f53\") " pod="openshift-ingress-canary/ingress-canary-99ktz" Jul 06 19:06:47.104063 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.103987 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ecd44d-1d0b-499b-9a05-a017cb70fa44-tmp-dir\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.104063 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.104004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4c8c169-88e1-4115-8387-7107d15cf4ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.104203 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.104126 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4c8c169-88e1-4115-8387-7107d15cf4ca-data-volume\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.104203 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.104156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ecd44d-1d0b-499b-9a05-a017cb70fa44-metrics-tls\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.104203 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.104188 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wz5\" (UniqueName: \"kubernetes.io/projected/36ecd44d-1d0b-499b-9a05-a017cb70fa44-kube-api-access-p9wz5\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.104328 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.104298 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36ecd44d-1d0b-499b-9a05-a017cb70fa44-tmp-dir\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.104481 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.104461 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4c8c169-88e1-4115-8387-7107d15cf4ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.104544 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.104459 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36ecd44d-1d0b-499b-9a05-a017cb70fa44-config-volume\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.107865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.107772 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4c8c169-88e1-4115-8387-7107d15cf4ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.107979 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.107811 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36ecd44d-1d0b-499b-9a05-a017cb70fa44-metrics-tls\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.108034 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.107991 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e66372-e151-4119-91da-bfdf10300f53-cert\") pod \"ingress-canary-99ktz\" (UID: \"10e66372-e151-4119-91da-bfdf10300f53\") " pod="openshift-ingress-canary/ingress-canary-99ktz" Jul 06 19:06:47.111921 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.111868 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wz5\" (UniqueName: \"kubernetes.io/projected/36ecd44d-1d0b-499b-9a05-a017cb70fa44-kube-api-access-p9wz5\") pod \"dns-default-kt26b\" (UID: \"36ecd44d-1d0b-499b-9a05-a017cb70fa44\") " pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.111921 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.111876 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwnt\" (UniqueName: \"kubernetes.io/projected/10e66372-e151-4119-91da-bfdf10300f53-kube-api-access-wpwnt\") pod \"ingress-canary-99ktz\" (UID: \"10e66372-e151-4119-91da-bfdf10300f53\") " pod="openshift-ingress-canary/ingress-canary-99ktz" Jul 06 19:06:47.112077 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.112051 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgg8\" (UniqueName: \"kubernetes.io/projected/c4c8c169-88e1-4115-8387-7107d15cf4ca-kube-api-access-vwgg8\") pod \"insights-runtime-extractor-2vbxq\" (UID: \"c4c8c169-88e1-4115-8387-7107d15cf4ca\") " pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.125968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.125946 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:47.149036 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.149018 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vbxq" Jul 06 19:06:47.168290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.168270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-99ktz" Jul 06 19:06:47.351233 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.351204 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kt26b"] Jul 06 19:06:47.354915 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.354888 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-99ktz"] Jul 06 19:06:47.355988 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:47.355956 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ecd44d_1d0b_499b_9a05_a017cb70fa44.slice/crio-bb867824d5a3298236f18190a0f5b4c2a928f608a54e00273aa5cd16f4153457 WatchSource:0}: Error finding container bb867824d5a3298236f18190a0f5b4c2a928f608a54e00273aa5cd16f4153457: Status 404 returned error can't find the container with id bb867824d5a3298236f18190a0f5b4c2a928f608a54e00273aa5cd16f4153457 Jul 06 19:06:47.358062 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.357881 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2vbxq"] Jul 06 19:06:47.359151 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.359113 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:47.359605 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:47.359571 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e66372_e151_4119_91da_bfdf10300f53.slice/crio-038f50fa60c6fd09a83449918781fc9747c531b2982b174d3a97992b606b66c7 WatchSource:0}: Error finding container 038f50fa60c6fd09a83449918781fc9747c531b2982b174d3a97992b606b66c7: Status 404 returned error can't find the container with id 038f50fa60c6fd09a83449918781fc9747c531b2982b174d3a97992b606b66c7 Jul 06 19:06:47.365586 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:47.365566 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c8c169_88e1_4115_8387_7107d15cf4ca.slice/crio-93feaf9d6dd01b2c935ef44d11b8fba39d58de4036a0f4d983b5675f9d6d4d13 WatchSource:0}: Error finding container 93feaf9d6dd01b2c935ef44d11b8fba39d58de4036a0f4d983b5675f9d6d4d13: Status 404 returned error can't find the container with id 93feaf9d6dd01b2c935ef44d11b8fba39d58de4036a0f4d983b5675f9d6d4d13 Jul 06 19:06:47.411153 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:47.411125 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:48.015327 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.015222 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:06:48.015327 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.015265 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:06:48.016037 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.015734 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:06:48.018731 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.018710 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:06:48.020556 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.020343 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:06:48.020556 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.020380 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-85brg\"" Jul 06 19:06:48.020715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.020620 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:06:48.021213 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.020770 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xsmh6\"" Jul 06 19:06:48.021213 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.020838 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:06:48.176300 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.176204 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kt26b" event={"ID":"36ecd44d-1d0b-499b-9a05-a017cb70fa44","Type":"ContainerStarted","Data":"bb867824d5a3298236f18190a0f5b4c2a928f608a54e00273aa5cd16f4153457"} Jul 06 19:06:48.178518 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.178444 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-99ktz" event={"ID":"10e66372-e151-4119-91da-bfdf10300f53","Type":"ContainerStarted","Data":"038f50fa60c6fd09a83449918781fc9747c531b2982b174d3a97992b606b66c7"} Jul 06 19:06:48.180057 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.179992 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vbxq" event={"ID":"c4c8c169-88e1-4115-8387-7107d15cf4ca","Type":"ContainerStarted","Data":"93feaf9d6dd01b2c935ef44d11b8fba39d58de4036a0f4d983b5675f9d6d4d13"} Jul 06 19:06:48.984881 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:48.984808 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:49.085914 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.085829 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:49.185092 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.185034 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:49.300342 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.300272 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:49.367629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.367592 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt"] Jul 06 19:06:49.372528 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.372482 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:49.378014 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.377993 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" Jul 06 19:06:49.381088 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.381068 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 19:06:49.381331 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.381087 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mwcl8\"" Jul 06 19:06:49.382902 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.382841 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt"] Jul 06 19:06:49.424177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.424153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bc9d731-ab5c-45eb-96e0-82cd518ba6df-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-spplt\" (UID: \"9bc9d731-ab5c-45eb-96e0-82cd518ba6df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" Jul 06 19:06:49.525440 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.525406 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bc9d731-ab5c-45eb-96e0-82cd518ba6df-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-spplt\" (UID: \"9bc9d731-ab5c-45eb-96e0-82cd518ba6df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" Jul 06 19:06:49.525611 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:49.525574 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 06 19:06:49.525683 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:06:49.525649 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bc9d731-ab5c-45eb-96e0-82cd518ba6df-tls-certificates podName:9bc9d731-ab5c-45eb-96e0-82cd518ba6df nodeName:}" failed. No retries permitted until 2026-07-06 19:06:50.025623918 +0000 UTC m=+37.566346127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9bc9d731-ab5c-45eb-96e0-82cd518ba6df-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-spplt" (UID: "9bc9d731-ab5c-45eb-96e0-82cd518ba6df") : secret "prometheus-operator-admission-webhook-tls" not found Jul 06 19:06:49.529988 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.529965 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:49.630242 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.630162 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:49.961450 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:49.961382 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:50.029945 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:50.029916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bc9d731-ab5c-45eb-96e0-82cd518ba6df-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-spplt\" (UID: \"9bc9d731-ab5c-45eb-96e0-82cd518ba6df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" Jul 06 19:06:50.033313 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:50.033096 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bc9d731-ab5c-45eb-96e0-82cd518ba6df-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-spplt\" (UID: \"9bc9d731-ab5c-45eb-96e0-82cd518ba6df\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" Jul 06 19:06:50.290670 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:50.290638 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" Jul 06 19:06:51.667408 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:51.667375 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:51.765907 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:51.765873 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:51.960616 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:51.960515 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:52.043899 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:52.043860 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:54.859980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:54.859899 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:54.960732 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:54.960649 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:55.013409 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:55.013384 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt"] Jul 06 19:06:55.016556 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:55.016527 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc9d731_ab5c_45eb_96e0_82cd518ba6df.slice/crio-7355095ee5d6a99555c6baa8c5aa726e64dec5a2f2fe8fc3e606b99f981b21c0 WatchSource:0}: Error finding container 7355095ee5d6a99555c6baa8c5aa726e64dec5a2f2fe8fc3e606b99f981b21c0: Status 404 returned error can't find the container with id 7355095ee5d6a99555c6baa8c5aa726e64dec5a2f2fe8fc3e606b99f981b21c0 Jul 06 19:06:55.198738 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:55.198697 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-99ktz" event={"ID":"10e66372-e151-4119-91da-bfdf10300f53","Type":"ContainerStarted","Data":"76584bcdaecfb53beda1bfd0458cf8f594438ba0692934ae465f4ae24a5b9b58"} Jul 06 19:06:55.201528 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:55.201493 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerStarted","Data":"116dad82562ac69a9f85afada8795fb5ae392d7ac775a01e92148239eb6a2660"} Jul 06 19:06:55.202956 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:55.202934 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kt26b" event={"ID":"36ecd44d-1d0b-499b-9a05-a017cb70fa44","Type":"ContainerStarted","Data":"c4e6331fc5dfd45d8829069658af3fff97acce39bcf62607919af2753d7a38cd"} Jul 06 19:06:55.203974 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:55.203958 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" event={"ID":"9bc9d731-ab5c-45eb-96e0-82cd518ba6df","Type":"ContainerStarted","Data":"7355095ee5d6a99555c6baa8c5aa726e64dec5a2f2fe8fc3e606b99f981b21c0"} Jul 06 19:06:55.215870 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:55.215833 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-99ktz" podStartSLOduration=1.689419715 podStartE2EDuration="9.215822699s" totalImagesPullingTime="7.526402984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:55.215027489 +0000 UTC m=+42.755749729" watchObservedRunningTime="2026-07-06 19:06:55.215822699 +0000 UTC m=+42.756544926" Jul 06 19:06:56.209668 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:56.209637 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca" containerID="116dad82562ac69a9f85afada8795fb5ae392d7ac775a01e92148239eb6a2660" exitCode=0 Jul 06 19:06:56.210090 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:56.209716 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerDied","Data":"116dad82562ac69a9f85afada8795fb5ae392d7ac775a01e92148239eb6a2660"} Jul 06 19:06:56.213990 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:56.213959 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vbxq" event={"ID":"c4c8c169-88e1-4115-8387-7107d15cf4ca","Type":"ContainerStarted","Data":"05512785807d1489c32cc527629f11d174e9fe8ff19ad0eddbc493245ce685a9"} Jul 06 19:06:56.216032 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:56.216006 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kt26b" event={"ID":"36ecd44d-1d0b-499b-9a05-a017cb70fa44","Type":"ContainerStarted","Data":"21e4bf4e44c1e8a51c85907c1c3cb506400133806201efb74c26b1f7d708f792"} Jul 06 19:06:56.216173 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:56.216155 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kt26b" Jul 06 19:06:56.253346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:56.253303 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kt26b" podStartSLOduration=2.724300326 podStartE2EDuration="10.253290563s" totalImagesPullingTime="7.528990237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:56.252547918 +0000 UTC m=+43.793270158" watchObservedRunningTime="2026-07-06 19:06:56.253290563 +0000 UTC m=+43.794012769" Jul 06 19:06:56.861442 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:56.861403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:56.962311 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:56.962290 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:57.224666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.224601 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca" containerID="7af234144d5d1f5406680711f2d67c800662bfff03340f596412631c8be733a5" exitCode=0 Jul 06 19:06:57.224993 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.224682 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerDied","Data":"7af234144d5d1f5406680711f2d67c800662bfff03340f596412631c8be733a5"} Jul 06 19:06:57.226484 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.226456 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vbxq" event={"ID":"c4c8c169-88e1-4115-8387-7107d15cf4ca","Type":"ContainerStarted","Data":"3d6305737add70eab85db3f92b665ad4e4cc6e237a9bca9189d49bf5588bb7b9"} Jul 06 19:06:57.227821 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.227801 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" event={"ID":"9bc9d731-ab5c-45eb-96e0-82cd518ba6df","Type":"ContainerStarted","Data":"80276a328f73081ff252db68a8742ee676f0a9f30b2122f0f745576c1035b08d"} Jul 06 19:06:57.243774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.243746 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" Jul 06 19:06:57.248171 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.248154 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" Jul 06 19:06:57.261726 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.261684 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-spplt" podStartSLOduration=6.529457859 podStartE2EDuration="8.261669112s" totalImagesPullingTime="1.732211253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:57.260472137 +0000 UTC m=+44.801194367" watchObservedRunningTime="2026-07-06 19:06:57.261669112 +0000 UTC m=+44.802391342" Jul 06 19:06:57.353070 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.353044 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:57.422494 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.422463 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:57.422600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.422477 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps"] Jul 06 19:06:57.444587 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.444567 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps"] Jul 06 19:06:57.444688 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.444661 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.447027 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.446987 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:06:57.447118 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.447073 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7xrlt\"" Jul 06 19:06:57.447118 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.447087 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:06:57.447230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.447075 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 19:06:57.447230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.447227 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 19:06:57.447479 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.447462 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:06:57.483764 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.483708 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.483764 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.483752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.483922 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.483785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mkj\" (UniqueName: \"kubernetes.io/projected/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-kube-api-access-g5mkj\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.483922 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.483864 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.489732 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.489704 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:57.585050 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.585023 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.585172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.585095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.585172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.585127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.585172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.585156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mkj\" (UniqueName: \"kubernetes.io/projected/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-kube-api-access-g5mkj\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.585838 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.585793 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.588198 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.588180 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.588329 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.588310 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.592973 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.592954 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mkj\" (UniqueName: \"kubernetes.io/projected/297fdc92-dd2d-45e8-81ee-8fb348b0b1c6-kube-api-access-g5mkj\") pod \"prometheus-operator-567f7b8b7c-bhrps\" (UID: \"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.754322 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.754256 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" Jul 06 19:06:57.907157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:57.907112 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps"] Jul 06 19:06:57.911184 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:06:57.911160 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297fdc92_dd2d_45e8_81ee_8fb348b0b1c6.slice/crio-ee35c5266df3e6db8060c9f0f4f8d53ec912be00446f70431fb65c7dea67fe0f WatchSource:0}: Error finding container ee35c5266df3e6db8060c9f0f4f8d53ec912be00446f70431fb65c7dea67fe0f: Status 404 returned error can't find the container with id ee35c5266df3e6db8060c9f0f4f8d53ec912be00446f70431fb65c7dea67fe0f Jul 06 19:06:58.232374 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:58.232334 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" event={"ID":"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6","Type":"ContainerStarted","Data":"ee35c5266df3e6db8060c9f0f4f8d53ec912be00446f70431fb65c7dea67fe0f"} Jul 06 19:06:58.235309 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:58.235284 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" event={"ID":"5cbfc4d9-77e2-4f59-b00d-9bd71a3f4fca","Type":"ContainerStarted","Data":"27711c90acf106635cb8905d72fd15920b3ffb5d9063cc743fac2d423cbb7ea4"} Jul 06 19:06:58.256276 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:58.256210 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rgpjd" podStartSLOduration=14.210028119 podStartE2EDuration="45.25619776s" totalImagesPullingTime="31.046169641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:58.255075965 +0000 UTC m=+45.795798193" watchObservedRunningTime="2026-07-06 19:06:58.25619776 +0000 UTC m=+45.796919979" Jul 06 19:06:58.377467 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:58.377390 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:58.441874 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:58.441844 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:58.673740 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:58.673703 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:06:58.734572 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:06:58.734527 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:00.244794 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:00.244469 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" event={"ID":"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6","Type":"ContainerStarted","Data":"9b1295dedbef9f575ef2ce54d7eb804d099cc2ddd1a627ff46df435a12a59a4f"} Jul 06 19:07:00.247035 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:00.247013 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vbxq" event={"ID":"c4c8c169-88e1-4115-8387-7107d15cf4ca","Type":"ContainerStarted","Data":"f204a513b749cec3a1831a52f07828580b2530d8bc812da061dc675ac28f02c3"} Jul 06 19:07:00.263402 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:00.263359 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2vbxq" podStartSLOduration=1.8240201969999998 podStartE2EDuration="14.263344231s" totalImagesPullingTime="12.439324034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:00.262847514 +0000 UTC m=+47.803569743" watchObservedRunningTime="2026-07-06 19:07:00.263344231 +0000 UTC m=+47.804066458" Jul 06 19:07:00.357772 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:00.357732 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:01.251897 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:01.251863 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" event={"ID":"297fdc92-dd2d-45e8-81ee-8fb348b0b1c6","Type":"ContainerStarted","Data":"97331afb6223688df331abd79f67dcb9d07aef42cdbe67e93601a21a848fa72c"} Jul 06 19:07:01.268502 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:01.268460 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-bhrps" podStartSLOduration=2.124658999 podStartE2EDuration="4.268445036s" totalImagesPullingTime="2.143786037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:01.26726608 +0000 UTC m=+48.807988310" watchObservedRunningTime="2026-07-06 19:07:01.268445036 +0000 UTC m=+48.809167256" Jul 06 19:07:02.354381 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.354348 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:02.410757 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.410732 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:02.529672 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.529627 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:02.756227 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.756151 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:02.770553 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.770529 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k"] Jul 06 19:07:02.772598 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.772584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.774802 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.774781 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 19:07:02.774802 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.774796 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:07:02.774936 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.774832 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lwkx8\"" Jul 06 19:07:02.785245 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.785223 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k"] Jul 06 19:07:02.795751 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.795729 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-szsdw"] Jul 06 19:07:02.797615 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.797595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.799946 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.799922 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:07:02.800035 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.799923 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:07:02.800111 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.800097 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:07:02.800290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.800278 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2xjff\"" Jul 06 19:07:02.824867 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.824844 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-root\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.824957 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.824878 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-tls\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.824957 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.824894 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8tq\" (UniqueName: \"kubernetes.io/projected/2382d583-1001-4a8d-8ed6-bffd91e5c90f-kube-api-access-8r8tq\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.824957 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.824944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-accelerators-collector-config\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.825055 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.824978 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-sys\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.825055 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.825012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.825055 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.825033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkkjl\" (UniqueName: \"kubernetes.io/projected/9a3ed9b2-92e1-4985-a593-ccdab305114b-kube-api-access-vkkjl\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.825055 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.825050 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-wtmp\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.825168 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.825090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.825168 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.825110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-textfile\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.825168 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.825140 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a3ed9b2-92e1-4985-a593-ccdab305114b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.825168 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.825163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.825279 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.825180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2382d583-1001-4a8d-8ed6-bffd91e5c90f-metrics-client-ca\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.842264 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.842241 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:02.911778 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.911756 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:02.925952 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.925931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-wtmp\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926015 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.925960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.926015 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.925976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-textfile\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926015 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926008 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a3ed9b2-92e1-4985-a593-ccdab305114b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.926125 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926034 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.926125 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2382d583-1001-4a8d-8ed6-bffd91e5c90f-metrics-client-ca\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926125 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:07:02.926121 2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 19:07:02.926247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-root\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926247 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:07:02.926167 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-tls podName:9a3ed9b2-92e1-4985-a593-ccdab305114b nodeName:}" failed. No retries permitted until 2026-07-06 19:07:03.426154467 +0000 UTC m=+50.966876674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-rbq6k" (UID: "9a3ed9b2-92e1-4985-a593-ccdab305114b") : secret "openshift-state-metrics-tls" not found Jul 06 19:07:02.926247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926189 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-root\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-tls\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926107 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-wtmp\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8tq\" (UniqueName: \"kubernetes.io/projected/2382d583-1001-4a8d-8ed6-bffd91e5c90f-kube-api-access-8r8tq\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926282 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-accelerators-collector-config\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-sys\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926336 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926367 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-textfile\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkkjl\" (UniqueName: \"kubernetes.io/projected/9a3ed9b2-92e1-4985-a593-ccdab305114b-kube-api-access-vkkjl\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.926629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926490 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2382d583-1001-4a8d-8ed6-bffd91e5c90f-sys\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926925 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926792 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2382d583-1001-4a8d-8ed6-bffd91e5c90f-metrics-client-ca\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.926925 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.926901 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a3ed9b2-92e1-4985-a593-ccdab305114b-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.927052 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.927025 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-accelerators-collector-config\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.928462 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.928440 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.928536 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.928468 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-tls\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.928574 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.928540 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2382d583-1001-4a8d-8ed6-bffd91e5c90f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.937247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.937224 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8tq\" (UniqueName: \"kubernetes.io/projected/2382d583-1001-4a8d-8ed6-bffd91e5c90f-kube-api-access-8r8tq\") pod \"node-exporter-szsdw\" (UID: \"2382d583-1001-4a8d-8ed6-bffd91e5c90f\") " pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:02.937547 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.937528 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkkjl\" (UniqueName: \"kubernetes.io/projected/9a3ed9b2-92e1-4985-a593-ccdab305114b-kube-api-access-vkkjl\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:02.992804 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:02.992775 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:03.047879 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.047854 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:03.108198 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.108175 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-szsdw" Jul 06 19:07:03.117284 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:07:03.117262 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2382d583_1001_4a8d_8ed6_bffd91e5c90f.slice/crio-826a8dc064793df6076e8d9784dd875ecf892d05089bffdebbf37fe7b32bac55 WatchSource:0}: Error finding container 826a8dc064793df6076e8d9784dd875ecf892d05089bffdebbf37fe7b32bac55: Status 404 returned error can't find the container with id 826a8dc064793df6076e8d9784dd875ecf892d05089bffdebbf37fe7b32bac55 Jul 06 19:07:03.198653 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.198626 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:03.259188 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.259165 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-szsdw" event={"ID":"2382d583-1001-4a8d-8ed6-bffd91e5c90f","Type":"ContainerStarted","Data":"826a8dc064793df6076e8d9784dd875ecf892d05089bffdebbf37fe7b32bac55"} Jul 06 19:07:03.298570 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.298509 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:03.399644 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.399612 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:03.431031 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.430994 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:03.433248 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.433229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a3ed9b2-92e1-4985-a593-ccdab305114b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rbq6k\" (UID: \"9a3ed9b2-92e1-4985-a593-ccdab305114b\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:03.499505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.499473 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:03.680440 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.680347 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" Jul 06 19:07:03.755239 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.755210 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:03.808130 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.808100 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k"] Jul 06 19:07:03.811925 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:07:03.811892 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3ed9b2_92e1_4985_a593_ccdab305114b.slice/crio-791114b9713f2b0a7625de3b412158b26d4c8eb82aeaf062aa420b44b87ea1f8 WatchSource:0}: Error finding container 791114b9713f2b0a7625de3b412158b26d4c8eb82aeaf062aa420b44b87ea1f8: Status 404 returned error can't find the container with id 791114b9713f2b0a7625de3b412158b26d4c8eb82aeaf062aa420b44b87ea1f8 Jul 06 19:07:03.839560 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:03.839533 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:04.263932 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:04.263911 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" event={"ID":"9a3ed9b2-92e1-4985-a593-ccdab305114b","Type":"ContainerStarted","Data":"69a610f007a0f34efd7a7c436ee0a8edcf4f42486a897d312cecf50b306ab6b8"} Jul 06 19:07:04.264010 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:04.263960 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" event={"ID":"9a3ed9b2-92e1-4985-a593-ccdab305114b","Type":"ContainerStarted","Data":"b2bba59dab6b9c7a5ab5e83da026d12ffadd7e1c7935908e2c82f6a29a5e4827"} Jul 06 19:07:04.264010 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:04.263981 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" event={"ID":"9a3ed9b2-92e1-4985-a593-ccdab305114b","Type":"ContainerStarted","Data":"791114b9713f2b0a7625de3b412158b26d4c8eb82aeaf062aa420b44b87ea1f8"} Jul 06 19:07:04.403646 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:04.403624 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:04.692348 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:04.692315 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:05.269187 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.269164 2569 generic.go:356] "Generic (PLEG): container finished" podID="2382d583-1001-4a8d-8ed6-bffd91e5c90f" containerID="8c749b427178409d8448ac9055561433cfc5fcece6d6424f090df1102cf564b4" exitCode=0 Jul 06 19:07:05.269286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.269235 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-szsdw" event={"ID":"2382d583-1001-4a8d-8ed6-bffd91e5c90f","Type":"ContainerDied","Data":"8c749b427178409d8448ac9055561433cfc5fcece6d6424f090df1102cf564b4"} Jul 06 19:07:05.271451 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.271410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" event={"ID":"9a3ed9b2-92e1-4985-a593-ccdab305114b","Type":"ContainerStarted","Data":"b07d0ceecf1f805b5656dfb44fa03ef3b3099d5951eee03b27eea941123e2d81"} Jul 06 19:07:05.319837 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.319799 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rbq6k" podStartSLOduration=2.043043557 podStartE2EDuration="3.31978479s" totalImagesPullingTime="1.276741233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:05.318783372 +0000 UTC m=+52.859505600" watchObservedRunningTime="2026-07-06 19:07:05.31978479 +0000 UTC m=+52.860507009" Jul 06 19:07:05.664744 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.664715 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:05.767803 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.767769 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:05.776622 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.776603 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-799df989cd-smgpk"] Jul 06 19:07:05.779034 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.779020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.781249 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.781226 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 19:07:05.781345 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.781290 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 19:07:05.781399 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.781340 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3li2pil7k8dp0\"" Jul 06 19:07:05.781531 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.781517 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 19:07:05.781957 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.781941 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 19:07:05.782034 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.782018 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m4t6m\"" Jul 06 19:07:05.782144 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.782128 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 19:07:05.790929 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.790908 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-799df989cd-smgpk"] Jul 06 19:07:05.831349 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.831326 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:05.853321 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.853298 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-grpc-tls\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.853405 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.853324 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-tls\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.853405 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.853343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xc22\" (UniqueName: \"kubernetes.io/projected/43a9777c-503c-4206-b4b6-768a95a28bc9-kube-api-access-6xc22\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.853405 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.853362 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.853530 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.853482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.853530 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.853517 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.853589 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.853538 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.853589 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.853581 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a9777c-503c-4206-b4b6-768a95a28bc9-metrics-client-ca\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.942237 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.942181 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:05.954070 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.954045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-grpc-tls\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.954155 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.954074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-tls\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.954155 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.954091 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xc22\" (UniqueName: \"kubernetes.io/projected/43a9777c-503c-4206-b4b6-768a95a28bc9-kube-api-access-6xc22\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.954155 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.954110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.954340 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.954318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.954387 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.954350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.954387 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.954377 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.954512 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.954444 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a9777c-503c-4206-b4b6-768a95a28bc9-metrics-client-ca\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.955254 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.955230 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43a9777c-503c-4206-b4b6-768a95a28bc9-metrics-client-ca\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.956887 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.956854 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.957086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.957065 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.957160 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.957137 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.957213 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.957176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-grpc-tls\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.957247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.957227 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.957279 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.957261 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/43a9777c-503c-4206-b4b6-768a95a28bc9-secret-thanos-querier-tls\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:05.961432 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:05.961401 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xc22\" (UniqueName: \"kubernetes.io/projected/43a9777c-503c-4206-b4b6-768a95a28bc9-kube-api-access-6xc22\") pod \"thanos-querier-799df989cd-smgpk\" (UID: \"43a9777c-503c-4206-b4b6-768a95a28bc9\") " pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:06.038079 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.038049 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:06.087457 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.087415 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:06.139481 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.139454 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:06.211830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.211769 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-799df989cd-smgpk"] Jul 06 19:07:06.214364 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:07:06.214339 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a9777c_503c_4206_b4b6_768a95a28bc9.slice/crio-ab9a2c7105ae07e21498487cc310b0a5043ba36dfb9b2c195348b8b772272a5b WatchSource:0}: Error finding container ab9a2c7105ae07e21498487cc310b0a5043ba36dfb9b2c195348b8b772272a5b: Status 404 returned error can't find the container with id ab9a2c7105ae07e21498487cc310b0a5043ba36dfb9b2c195348b8b772272a5b Jul 06 19:07:06.245840 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.245821 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kt26b" Jul 06 19:07:06.275948 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.275925 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" event={"ID":"43a9777c-503c-4206-b4b6-768a95a28bc9","Type":"ContainerStarted","Data":"ab9a2c7105ae07e21498487cc310b0a5043ba36dfb9b2c195348b8b772272a5b"} Jul 06 19:07:06.278123 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.278100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-szsdw" event={"ID":"2382d583-1001-4a8d-8ed6-bffd91e5c90f","Type":"ContainerStarted","Data":"0af1977b8ec94aaaa7444e1b1163ea0c8ba354fb76426e26e0810139f367b311"} Jul 06 19:07:06.278217 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.278144 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-szsdw" event={"ID":"2382d583-1001-4a8d-8ed6-bffd91e5c90f","Type":"ContainerStarted","Data":"7f4452b75093fffd2153c3c4cbc2614997ca7f76166fbdfefe25194957aa5b3b"} Jul 06 19:07:06.299029 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:06.298992 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-szsdw" podStartSLOduration=3.121617968 podStartE2EDuration="4.29897917s" totalImagesPullingTime="1.115295931s" totalInitContainerRuntime="62.065271ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:06.298404491 +0000 UTC m=+53.839126720" watchObservedRunningTime="2026-07-06 19:07:06.29897917 +0000 UTC m=+53.839701398" Jul 06 19:07:07.296111 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.296080 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:07.348764 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.348733 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:07.576763 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.574335 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2"] Jul 06 19:07:07.577765 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.577633 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:07.580080 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.580036 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 19:07:07.580080 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.580051 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gx6cf\"" Jul 06 19:07:07.580485 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.580462 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:07.587708 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.587651 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2"] Jul 06 19:07:07.641873 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.641850 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:07.667948 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.667925 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8d5de75-d3ef-43be-8930-b2af5823b6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-96bl2\" (UID: \"a8d5de75-d3ef-43be-8930-b2af5823b6ab\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:07.668077 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.667956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26j2x\" (UniqueName: \"kubernetes.io/projected/a8d5de75-d3ef-43be-8930-b2af5823b6ab-kube-api-access-26j2x\") pod \"monitoring-plugin-d5dcff7cf-96bl2\" (UID: \"a8d5de75-d3ef-43be-8930-b2af5823b6ab\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:07.769274 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.769239 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8d5de75-d3ef-43be-8930-b2af5823b6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-96bl2\" (UID: \"a8d5de75-d3ef-43be-8930-b2af5823b6ab\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:07.769444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.769285 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26j2x\" (UniqueName: \"kubernetes.io/projected/a8d5de75-d3ef-43be-8930-b2af5823b6ab-kube-api-access-26j2x\") pod \"monitoring-plugin-d5dcff7cf-96bl2\" (UID: \"a8d5de75-d3ef-43be-8930-b2af5823b6ab\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:07.769444 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:07:07.769420 2569 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 19:07:07.769559 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:07:07.769496 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8d5de75-d3ef-43be-8930-b2af5823b6ab-monitoring-plugin-cert podName:a8d5de75-d3ef-43be-8930-b2af5823b6ab nodeName:}" failed. No retries permitted until 2026-07-06 19:07:08.269481159 +0000 UTC m=+55.810203376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a8d5de75-d3ef-43be-8930-b2af5823b6ab-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-96bl2" (UID: "a8d5de75-d3ef-43be-8930-b2af5823b6ab") : secret "monitoring-plugin-cert" not found Jul 06 19:07:07.779880 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:07.779852 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26j2x\" (UniqueName: \"kubernetes.io/projected/a8d5de75-d3ef-43be-8930-b2af5823b6ab-kube-api-access-26j2x\") pod \"monitoring-plugin-d5dcff7cf-96bl2\" (UID: \"a8d5de75-d3ef-43be-8930-b2af5823b6ab\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:08.011557 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.011509 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-859499568-8r4sz"] Jul 06 19:07:08.014359 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.014327 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.016798 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.016780 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 19:07:08.016904 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.016868 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 19:07:08.016976 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.016917 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zqhng\"" Jul 06 19:07:08.016976 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.016928 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 19:07:08.016976 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.016940 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 19:07:08.017240 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.017224 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 19:07:08.022717 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.022595 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 19:07:08.027709 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.027690 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-859499568-8r4sz"] Jul 06 19:07:08.071861 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.071833 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgst\" (UniqueName: \"kubernetes.io/projected/ad8bfb29-5f89-495a-a3db-5962a3e89124-kube-api-access-4fgst\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.072004 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.071957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-secret-telemeter-client\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.072069 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.072013 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.072069 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.072056 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-federate-client-tls\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.072152 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.072087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-telemeter-trusted-ca-bundle\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.072152 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.072146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-serving-certs-ca-bundle\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.072229 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.072179 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-telemeter-client-tls\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.072274 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.072229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-metrics-client-ca\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.174157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.173890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-secret-telemeter-client\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.174157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.173943 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.174157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.173976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-federate-client-tls\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.174157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.174001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-telemeter-trusted-ca-bundle\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.174157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.174022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-serving-certs-ca-bundle\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.174157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.174049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-telemeter-client-tls\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.174157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.174091 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-metrics-client-ca\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.174157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.174131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgst\" (UniqueName: \"kubernetes.io/projected/ad8bfb29-5f89-495a-a3db-5962a3e89124-kube-api-access-4fgst\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.175380 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.175113 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-serving-certs-ca-bundle\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.175380 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.175280 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-telemeter-trusted-ca-bundle\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.175548 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.175456 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad8bfb29-5f89-495a-a3db-5962a3e89124-metrics-client-ca\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.177316 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.177265 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.177746 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.177725 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-federate-client-tls\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.177828 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.177773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-telemeter-client-tls\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.178468 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.178448 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ad8bfb29-5f89-495a-a3db-5962a3e89124-secret-telemeter-client\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.181645 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.181628 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgst\" (UniqueName: \"kubernetes.io/projected/ad8bfb29-5f89-495a-a3db-5962a3e89124-kube-api-access-4fgst\") pod \"telemeter-client-859499568-8r4sz\" (UID: \"ad8bfb29-5f89-495a-a3db-5962a3e89124\") " pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.215460 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.215416 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:08.274575 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.274548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8d5de75-d3ef-43be-8930-b2af5823b6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-96bl2\" (UID: \"a8d5de75-d3ef-43be-8930-b2af5823b6ab\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:08.276915 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.276892 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8d5de75-d3ef-43be-8930-b2af5823b6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-96bl2\" (UID: \"a8d5de75-d3ef-43be-8930-b2af5823b6ab\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:08.287141 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.287117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" event={"ID":"43a9777c-503c-4206-b4b6-768a95a28bc9","Type":"ContainerStarted","Data":"2a6b2252481a7361cb60bc6e47e4de6dfdfa7149cd085841c1d69b91ecffe579"} Jul 06 19:07:08.287248 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.287184 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" event={"ID":"43a9777c-503c-4206-b4b6-768a95a28bc9","Type":"ContainerStarted","Data":"469aa3945dbd48c12fa0f8bde2d0f01ccc8fb0ca53a41fe06b20bc6c2e90a39f"} Jul 06 19:07:08.325511 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.325485 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-859499568-8r4sz" Jul 06 19:07:08.364135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.364107 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:08.446867 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.446736 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-859499568-8r4sz"] Jul 06 19:07:08.449197 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:07:08.449174 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8bfb29_5f89_495a_a3db_5962a3e89124.slice/crio-41da7f850ab48ef48c59871da86de3c9ef4dca5315cc46b6dd78a7c95f1a4852 WatchSource:0}: Error finding container 41da7f850ab48ef48c59871da86de3c9ef4dca5315cc46b6dd78a7c95f1a4852: Status 404 returned error can't find the container with id 41da7f850ab48ef48c59871da86de3c9ef4dca5315cc46b6dd78a7c95f1a4852 Jul 06 19:07:08.449510 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.449482 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:08.489519 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.489496 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:08.609468 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:08.609442 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2"] Jul 06 19:07:08.612652 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:07:08.612626 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d5de75_d3ef_43be_8930_b2af5823b6ab.slice/crio-9f05f8e9b35c45609897c74bd23566a31e5f4ae1be5a22d27da4c5e38c90fc09 WatchSource:0}: Error finding container 9f05f8e9b35c45609897c74bd23566a31e5f4ae1be5a22d27da4c5e38c90fc09: Status 404 returned error can't find the container with id 9f05f8e9b35c45609897c74bd23566a31e5f4ae1be5a22d27da4c5e38c90fc09 Jul 06 19:07:09.006105 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.006022 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:09.281877 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.281850 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:09.292158 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.292129 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-859499568-8r4sz" event={"ID":"ad8bfb29-5f89-495a-a3db-5962a3e89124","Type":"ContainerStarted","Data":"41da7f850ab48ef48c59871da86de3c9ef4dca5315cc46b6dd78a7c95f1a4852"} Jul 06 19:07:09.294687 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.294667 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" event={"ID":"43a9777c-503c-4206-b4b6-768a95a28bc9","Type":"ContainerStarted","Data":"3e820d5d05767003eae47aadb92c20f9802309b4e61567a881a8cba25fbbfe98"} Jul 06 19:07:09.294787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.294736 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" event={"ID":"43a9777c-503c-4206-b4b6-768a95a28bc9","Type":"ContainerStarted","Data":"08e94b62b4520f8d55672a619558f2f05c6151dc9dbb3300fd5c25ca57ee9185"} Jul 06 19:07:09.297912 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.297892 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" event={"ID":"a8d5de75-d3ef-43be-8930-b2af5823b6ab","Type":"ContainerStarted","Data":"9f05f8e9b35c45609897c74bd23566a31e5f4ae1be5a22d27da4c5e38c90fc09"} Jul 06 19:07:09.380261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.380194 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:09.486027 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.485990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:09.581401 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.581336 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:09.704680 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.704643 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:09.806281 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:09.806241 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:10.304548 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:10.304504 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" event={"ID":"43a9777c-503c-4206-b4b6-768a95a28bc9","Type":"ContainerStarted","Data":"e1948d9a7fa89d373fb59b446ea766606505deebce51b0cdf405260e746d6e22"} Jul 06 19:07:10.304718 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:10.304617 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" event={"ID":"43a9777c-503c-4206-b4b6-768a95a28bc9","Type":"ContainerStarted","Data":"53cb0bf78244a955d43dedae5d0211d7bafc1307884e4c2106c208337c04ef5b"} Jul 06 19:07:10.304718 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:10.304669 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:10.306090 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:10.306070 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" event={"ID":"a8d5de75-d3ef-43be-8930-b2af5823b6ab","Type":"ContainerStarted","Data":"de449c6e5b94b3c01782dad49415da48869925f91ee2da468b9b2229deb7c5b1"} Jul 06 19:07:10.306453 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:10.306433 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:10.311503 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:10.311484 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" Jul 06 19:07:10.345627 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:10.345587 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" podStartSLOduration=2.559382117 podStartE2EDuration="5.345574142s" totalImagesPullingTime="2.786192025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:10.343171694 +0000 UTC m=+57.883893922" watchObservedRunningTime="2026-07-06 19:07:10.345574142 +0000 UTC m=+57.886296368" Jul 06 19:07:10.366231 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:10.366149 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-96bl2" podStartSLOduration=1.8815550540000001 podStartE2EDuration="3.366136338s" totalImagesPullingTime="1.484581284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:10.365640831 +0000 UTC m=+57.906363060" watchObservedRunningTime="2026-07-06 19:07:10.366136338 +0000 UTC m=+57.906858575" Jul 06 19:07:11.310741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:11.310709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-859499568-8r4sz" event={"ID":"ad8bfb29-5f89-495a-a3db-5962a3e89124","Type":"ContainerStarted","Data":"b851daa6954574b001ffe51436a068f69fe51209e405eee2a25f06cc38cb57e8"} Jul 06 19:07:12.316617 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:12.316578 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-859499568-8r4sz" event={"ID":"ad8bfb29-5f89-495a-a3db-5962a3e89124","Type":"ContainerStarted","Data":"e247b9e8d9eada404d695dc8851589f43739cec7527f67c0542e9fa5b684a271"} Jul 06 19:07:12.316949 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:12.316671 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-859499568-8r4sz" event={"ID":"ad8bfb29-5f89-495a-a3db-5962a3e89124","Type":"ContainerStarted","Data":"3b6f6dcc2dedbdf4a56b6505f69f02af918db00b6e61b8332cdb42d401d9518e"} Jul 06 19:07:12.343175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:12.343097 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-859499568-8r4sz" podStartSLOduration=1.780505099 podStartE2EDuration="5.343082813s" totalImagesPullingTime="3.562577714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:12.341365428 +0000 UTC m=+59.882087659" watchObservedRunningTime="2026-07-06 19:07:12.343082813 +0000 UTC m=+59.883805041" Jul 06 19:07:13.171386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:13.171361 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2lch" Jul 06 19:07:13.192049 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:13.192014 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:13.291621 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:13.291566 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:13.388109 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:13.388073 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:13.491630 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:13.491544 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:15.363097 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:15.363069 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:15.431130 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:15.431103 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:16.315903 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:16.315878 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-799df989cd-smgpk" Jul 06 19:07:16.461884 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:16.461845 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:16.565503 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:16.565473 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:17.362563 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:17.362523 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:17.462463 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:17.462418 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:17.571296 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:17.571268 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:18.657809 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.657777 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:07:18.657809 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.657813 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:07:18.660530 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.660512 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:07:18.661277 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.661262 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:07:18.670123 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.670100 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8b29103-52ce-44a9-9d65-8c55bb83ed04-metrics-certs\") pod \"network-metrics-daemon-cwsgf\" (UID: \"a8b29103-52ce-44a9-9d65-8c55bb83ed04\") " pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:07:18.670510 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.670496 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:07:18.680257 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.680234 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvcgn\" (UniqueName: \"kubernetes.io/projected/d8c1f921-195b-46c7-8245-02e16d2679d7-kube-api-access-wvcgn\") pod \"network-check-target-tfvld\" (UID: \"d8c1f921-195b-46c7-8245-02e16d2679d7\") " pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:07:18.759086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.759051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:07:18.761417 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.761401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:07:18.771225 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.771200 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09a32004-05e2-4a14-9943-7b9b3229cf2f-original-pull-secret\") pod \"global-pull-secret-syncer-fmm9q\" (UID: \"09a32004-05e2-4a14-9943-7b9b3229cf2f\") " pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:07:18.932043 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.931974 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fmm9q" Jul 06 19:07:18.943540 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.943516 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-85brg\"" Jul 06 19:07:18.950830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.950809 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xsmh6\"" Jul 06 19:07:18.951794 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.951778 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cwsgf" Jul 06 19:07:18.958737 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:18.958721 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:07:19.096975 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:19.096845 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fmm9q"] Jul 06 19:07:19.099835 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:07:19.099811 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a32004_05e2_4a14_9943_7b9b3229cf2f.slice/crio-5c2f59d61fcb7ec6974891c766c9bb965db5077006e82f50da3b7537f479bf51 WatchSource:0}: Error finding container 5c2f59d61fcb7ec6974891c766c9bb965db5077006e82f50da3b7537f479bf51: Status 404 returned error can't find the container with id 5c2f59d61fcb7ec6974891c766c9bb965db5077006e82f50da3b7537f479bf51 Jul 06 19:07:19.328180 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:19.328148 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tfvld"] Jul 06 19:07:19.329198 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:19.329177 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cwsgf"] Jul 06 19:07:19.331550 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:07:19.331523 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c1f921_195b_46c7_8245_02e16d2679d7.slice/crio-e6a8c97c26025110bf6f5599b6f846939d20043d5cb2fc2a81418dcd3327e29e WatchSource:0}: Error finding container e6a8c97c26025110bf6f5599b6f846939d20043d5cb2fc2a81418dcd3327e29e: Status 404 returned error can't find the container with id e6a8c97c26025110bf6f5599b6f846939d20043d5cb2fc2a81418dcd3327e29e Jul 06 19:07:19.332109 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:07:19.332089 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b29103_52ce_44a9_9d65_8c55bb83ed04.slice/crio-f40c1e75bd4589e2c89e39d1958898cc90b282b45dba64753b716d738a4ed5bd WatchSource:0}: Error finding container f40c1e75bd4589e2c89e39d1958898cc90b282b45dba64753b716d738a4ed5bd: Status 404 returned error can't find the container with id f40c1e75bd4589e2c89e39d1958898cc90b282b45dba64753b716d738a4ed5bd Jul 06 19:07:19.345037 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:19.345014 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tfvld" event={"ID":"d8c1f921-195b-46c7-8245-02e16d2679d7","Type":"ContainerStarted","Data":"e6a8c97c26025110bf6f5599b6f846939d20043d5cb2fc2a81418dcd3327e29e"} Jul 06 19:07:19.346235 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:19.346210 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cwsgf" event={"ID":"a8b29103-52ce-44a9-9d65-8c55bb83ed04","Type":"ContainerStarted","Data":"f40c1e75bd4589e2c89e39d1958898cc90b282b45dba64753b716d738a4ed5bd"} Jul 06 19:07:19.347192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:19.347174 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fmm9q" event={"ID":"09a32004-05e2-4a14-9943-7b9b3229cf2f","Type":"ContainerStarted","Data":"5c2f59d61fcb7ec6974891c766c9bb965db5077006e82f50da3b7537f479bf51"} Jul 06 19:07:20.386805 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:20.386731 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:20.486474 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:20.486420 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:21.360925 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:21.360881 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cwsgf" event={"ID":"a8b29103-52ce-44a9-9d65-8c55bb83ed04","Type":"ContainerStarted","Data":"f921b758c1ea9a17a68c9f8e6b5973be848cdb59861092913ab21cbd77267010"} Jul 06 19:07:21.361102 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:21.360945 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cwsgf" event={"ID":"a8b29103-52ce-44a9-9d65-8c55bb83ed04","Type":"ContainerStarted","Data":"abbff42a3a9c73172b50c79eed70f6d188cdfcead24340443c5a46cf72166b36"} Jul 06 19:07:21.381165 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:21.381118 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cwsgf" podStartSLOduration=67.078317939 podStartE2EDuration="1m8.381098299s" totalImagesPullingTime="1.30278036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:21.381027973 +0000 UTC m=+68.921750192" watchObservedRunningTime="2026-07-06 19:07:21.381098299 +0000 UTC m=+68.921820527" Jul 06 19:07:21.490217 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:21.490187 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:21.554388 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:21.554358 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:21.617446 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:21.617331 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:24.375391 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:24.375354 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tfvld" event={"ID":"d8c1f921-195b-46c7-8245-02e16d2679d7","Type":"ContainerStarted","Data":"20c112d6ecc449b7cb1711c41e2ec3194874e96c2eb067570c8d97f11510bff2"} Jul 06 19:07:24.375826 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:24.375479 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:07:24.376906 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:24.376883 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fmm9q" event={"ID":"09a32004-05e2-4a14-9943-7b9b3229cf2f","Type":"ContainerStarted","Data":"74340103f7260eff6b9263115b61d69f0e41d88d04219e48b327b73de407635c"} Jul 06 19:07:24.393633 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:24.393593 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tfvld" podStartSLOduration=66.806267376 podStartE2EDuration="1m11.393581856s" totalImagesPullingTime="4.58731448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:24.391941579 +0000 UTC m=+71.932663799" watchObservedRunningTime="2026-07-06 19:07:24.393581856 +0000 UTC m=+71.934304084" Jul 06 19:07:24.407344 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:24.407304 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fmm9q" podStartSLOduration=66.58528644 podStartE2EDuration="1m11.407293676s" totalImagesPullingTime="4.822007236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:24.406549606 +0000 UTC m=+71.947271835" watchObservedRunningTime="2026-07-06 19:07:24.407293676 +0000 UTC m=+71.948015903" Jul 06 19:07:24.536771 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:24.536748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:24.637574 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:24.637507 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:24.694307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:24.694285 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:32.363062 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:32.363032 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:32.427585 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:32.427560 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:45.487401 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:45.487262 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:45.549684 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:45.549650 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:47.366557 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:47.366515 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:47.447758 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:47.447720 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:51.715246 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:51.715213 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:07:55.382044 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:07:55.382007 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tfvld" Jul 06 19:08:09.737481 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:09.737446 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:09.804971 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:09.804947 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:18.895999 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:18.895963 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:18.994589 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:18.994558 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:30.731779 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:30.731728 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:30.805556 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:30.805525 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:31.507205 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:31.507173 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:31.607732 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:31.607701 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:42.828146 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:42.828110 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:42.889314 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:42.889265 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:42.953090 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:42.953061 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:47.440692 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:47.440647 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:08:47.538504 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:08:47.538477 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:08.386136 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:08.386051 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:08.482474 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:08.482438 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:09.451991 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.451958 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk"] Jul 06 19:09:09.454492 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.454470 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.458215 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.458191 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:09:09.458450 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.458418 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nhb6z\"" Jul 06 19:09:09.458856 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.458842 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:09:09.465002 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.464981 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk"] Jul 06 19:09:09.502214 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.502195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.502311 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.502240 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7xb5\" (UniqueName: \"kubernetes.io/projected/dffe457e-fe54-4b16-94b2-326b9f6d85de-kube-api-access-p7xb5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.502311 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.502260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.603484 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.603459 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7xb5\" (UniqueName: \"kubernetes.io/projected/dffe457e-fe54-4b16-94b2-326b9f6d85de-kube-api-access-p7xb5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.603579 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.603487 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.603579 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.603517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.603798 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.603780 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.603834 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.603822 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.612784 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.612755 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7xb5\" (UniqueName: \"kubernetes.io/projected/dffe457e-fe54-4b16-94b2-326b9f6d85de-kube-api-access-p7xb5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.763314 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.763294 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:09.878485 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:09.878463 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk"] Jul 06 19:09:09.880985 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:09:09.880958 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffe457e_fe54_4b16_94b2_326b9f6d85de.slice/crio-d44bfd4a93a7d9cb593cfe4ea5cb8ec4435eb2949a3cd86d51dbd4bb65e847ef WatchSource:0}: Error finding container d44bfd4a93a7d9cb593cfe4ea5cb8ec4435eb2949a3cd86d51dbd4bb65e847ef: Status 404 returned error can't find the container with id d44bfd4a93a7d9cb593cfe4ea5cb8ec4435eb2949a3cd86d51dbd4bb65e847ef Jul 06 19:09:10.290153 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:10.290086 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:10.736301 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:10.736225 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" event={"ID":"dffe457e-fe54-4b16-94b2-326b9f6d85de","Type":"ContainerStarted","Data":"d44bfd4a93a7d9cb593cfe4ea5cb8ec4435eb2949a3cd86d51dbd4bb65e847ef"} Jul 06 19:09:11.693859 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:11.693818 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:19.775649 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:19.775613 2569 generic.go:356] "Generic (PLEG): container finished" podID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerID="96b41bc40d4e16e6367611b9312514fd342efd4144dc639bfa3e3754c5169624" exitCode=0 Jul 06 19:09:19.776013 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:19.775698 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" event={"ID":"dffe457e-fe54-4b16-94b2-326b9f6d85de","Type":"ContainerDied","Data":"96b41bc40d4e16e6367611b9312514fd342efd4144dc639bfa3e3754c5169624"} Jul 06 19:09:21.785159 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:21.785072 2569 generic.go:356] "Generic (PLEG): container finished" podID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerID="95d5f72c7e007f52920e2a59b487208d02d8cacff3fa0c5a4c9f14ff36415c84" exitCode=0 Jul 06 19:09:21.785159 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:21.785118 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" event={"ID":"dffe457e-fe54-4b16-94b2-326b9f6d85de","Type":"ContainerDied","Data":"95d5f72c7e007f52920e2a59b487208d02d8cacff3fa0c5a4c9f14ff36415c84"} Jul 06 19:09:31.831742 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:31.831712 2569 generic.go:356] "Generic (PLEG): container finished" podID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerID="cd75b5371cd510a160e0c1bbfff9d4852dcfb2f2d0c0d73b679a9c09e556cba4" exitCode=0 Jul 06 19:09:31.832099 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:31.831795 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" event={"ID":"dffe457e-fe54-4b16-94b2-326b9f6d85de","Type":"ContainerDied","Data":"cd75b5371cd510a160e0c1bbfff9d4852dcfb2f2d0c0d73b679a9c09e556cba4"} Jul 06 19:09:32.971040 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:32.971017 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:33.096015 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.095943 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-util\") pod \"dffe457e-fe54-4b16-94b2-326b9f6d85de\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " Jul 06 19:09:33.096163 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.096102 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-bundle\") pod \"dffe457e-fe54-4b16-94b2-326b9f6d85de\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " Jul 06 19:09:33.096163 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.096144 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7xb5\" (UniqueName: \"kubernetes.io/projected/dffe457e-fe54-4b16-94b2-326b9f6d85de-kube-api-access-p7xb5\") pod \"dffe457e-fe54-4b16-94b2-326b9f6d85de\" (UID: \"dffe457e-fe54-4b16-94b2-326b9f6d85de\") " Jul 06 19:09:33.096785 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.096762 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-bundle" (OuterVolumeSpecName: "bundle") pod "dffe457e-fe54-4b16-94b2-326b9f6d85de" (UID: "dffe457e-fe54-4b16-94b2-326b9f6d85de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:09:33.098200 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.098176 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffe457e-fe54-4b16-94b2-326b9f6d85de-kube-api-access-p7xb5" (OuterVolumeSpecName: "kube-api-access-p7xb5") pod "dffe457e-fe54-4b16-94b2-326b9f6d85de" (UID: "dffe457e-fe54-4b16-94b2-326b9f6d85de"). InnerVolumeSpecName "kube-api-access-p7xb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:09:33.100138 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.100113 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-util" (OuterVolumeSpecName: "util") pod "dffe457e-fe54-4b16-94b2-326b9f6d85de" (UID: "dffe457e-fe54-4b16-94b2-326b9f6d85de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:09:33.197206 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.197179 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-bundle\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:09:33.197206 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.197202 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7xb5\" (UniqueName: \"kubernetes.io/projected/dffe457e-fe54-4b16-94b2-326b9f6d85de-kube-api-access-p7xb5\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:09:33.197206 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.197214 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dffe457e-fe54-4b16-94b2-326b9f6d85de-util\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:09:33.841629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.841600 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" Jul 06 19:09:33.841629 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.841612 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dzmwk" event={"ID":"dffe457e-fe54-4b16-94b2-326b9f6d85de","Type":"ContainerDied","Data":"d44bfd4a93a7d9cb593cfe4ea5cb8ec4435eb2949a3cd86d51dbd4bb65e847ef"} Jul 06 19:09:33.841814 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:33.841650 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44bfd4a93a7d9cb593cfe4ea5cb8ec4435eb2949a3cd86d51dbd4bb65e847ef" Jul 06 19:09:35.538288 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:35.538257 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:36.566648 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:36.566605 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:37.436333 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.436294 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8"] Jul 06 19:09:37.436655 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.436638 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerName="pull" Jul 06 19:09:37.436733 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.436658 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerName="pull" Jul 06 19:09:37.436733 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.436675 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerName="util" Jul 06 19:09:37.436733 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.436683 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerName="util" Jul 06 19:09:37.436733 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.436707 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerName="extract" Jul 06 19:09:37.436733 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.436715 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerName="extract" Jul 06 19:09:37.436964 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.436785 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dffe457e-fe54-4b16-94b2-326b9f6d85de" containerName="extract" Jul 06 19:09:37.439492 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.439471 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" Jul 06 19:09:37.441819 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.441796 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:09:37.442026 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.442009 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 19:09:37.442073 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.442025 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vsvsl\"" Jul 06 19:09:37.454025 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.454008 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8"] Jul 06 19:09:37.628472 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.628444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4tl9\" (UniqueName: \"kubernetes.io/projected/000c689c-6fb5-4f8c-9d74-1a2896369d1b-kube-api-access-n4tl9\") pod \"cert-manager-operator-controller-manager-55c444bf86-2znc8\" (UID: \"000c689c-6fb5-4f8c-9d74-1a2896369d1b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" Jul 06 19:09:37.628806 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.628495 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/000c689c-6fb5-4f8c-9d74-1a2896369d1b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2znc8\" (UID: \"000c689c-6fb5-4f8c-9d74-1a2896369d1b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" Jul 06 19:09:37.729142 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.729079 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4tl9\" (UniqueName: \"kubernetes.io/projected/000c689c-6fb5-4f8c-9d74-1a2896369d1b-kube-api-access-n4tl9\") pod \"cert-manager-operator-controller-manager-55c444bf86-2znc8\" (UID: \"000c689c-6fb5-4f8c-9d74-1a2896369d1b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" Jul 06 19:09:37.729142 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.729129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/000c689c-6fb5-4f8c-9d74-1a2896369d1b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2znc8\" (UID: \"000c689c-6fb5-4f8c-9d74-1a2896369d1b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" Jul 06 19:09:37.729454 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.729419 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/000c689c-6fb5-4f8c-9d74-1a2896369d1b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2znc8\" (UID: \"000c689c-6fb5-4f8c-9d74-1a2896369d1b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" Jul 06 19:09:37.740370 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.740344 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4tl9\" (UniqueName: \"kubernetes.io/projected/000c689c-6fb5-4f8c-9d74-1a2896369d1b-kube-api-access-n4tl9\") pod \"cert-manager-operator-controller-manager-55c444bf86-2znc8\" (UID: \"000c689c-6fb5-4f8c-9d74-1a2896369d1b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" Jul 06 19:09:37.748180 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.748162 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" Jul 06 19:09:37.870229 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:37.870209 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8"] Jul 06 19:09:37.872861 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:09:37.872831 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000c689c_6fb5_4f8c_9d74_1a2896369d1b.slice/crio-b35845dc0f96d3f3f51bcca0497e11f0b7257748c013aa13e5d80611171a52ab WatchSource:0}: Error finding container b35845dc0f96d3f3f51bcca0497e11f0b7257748c013aa13e5d80611171a52ab: Status 404 returned error can't find the container with id b35845dc0f96d3f3f51bcca0497e11f0b7257748c013aa13e5d80611171a52ab Jul 06 19:09:38.862160 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:38.862121 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" event={"ID":"000c689c-6fb5-4f8c-9d74-1a2896369d1b","Type":"ContainerStarted","Data":"b35845dc0f96d3f3f51bcca0497e11f0b7257748c013aa13e5d80611171a52ab"} Jul 06 19:09:39.350824 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:39.350794 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:39.424449 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:39.424406 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:39.867522 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:39.867489 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" event={"ID":"000c689c-6fb5-4f8c-9d74-1a2896369d1b","Type":"ContainerStarted","Data":"1bb6da6648a2e79d4ac587625a7c6d4748788f9fbfea3f90c9bd56b6b0e7c828"} Jul 06 19:09:39.898342 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:39.898300 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2znc8" podStartSLOduration=1.273880733 podStartE2EDuration="2.898287308s" totalImagesPullingTime="1.624406575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:39.897674326 +0000 UTC m=+207.438396558" watchObservedRunningTime="2026-07-06 19:09:39.898287308 +0000 UTC m=+207.439009535" Jul 06 19:09:40.307946 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:40.307871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:40.513785 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:40.513737 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:40.879122 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:40.879089 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7xsfc"] Jul 06 19:09:40.881794 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:40.881776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:40.884102 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:40.884086 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:09:40.884391 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:40.884378 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:09:40.884947 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:40.884936 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7t59g\"" Jul 06 19:09:40.897539 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:40.897521 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7xsfc"] Jul 06 19:09:41.056173 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.056147 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z89zj\" (UniqueName: \"kubernetes.io/projected/b4dc2942-fb5d-473b-93af-c02d761e2d5d-kube-api-access-z89zj\") pod \"cert-manager-webhook-6486ffd997-7xsfc\" (UID: \"b4dc2942-fb5d-473b-93af-c02d761e2d5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:41.056340 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.056196 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4dc2942-fb5d-473b-93af-c02d761e2d5d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7xsfc\" (UID: \"b4dc2942-fb5d-473b-93af-c02d761e2d5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:41.156959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.156881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4dc2942-fb5d-473b-93af-c02d761e2d5d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7xsfc\" (UID: \"b4dc2942-fb5d-473b-93af-c02d761e2d5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:41.157099 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.156991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z89zj\" (UniqueName: \"kubernetes.io/projected/b4dc2942-fb5d-473b-93af-c02d761e2d5d-kube-api-access-z89zj\") pod \"cert-manager-webhook-6486ffd997-7xsfc\" (UID: \"b4dc2942-fb5d-473b-93af-c02d761e2d5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:41.167852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.167821 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4dc2942-fb5d-473b-93af-c02d761e2d5d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7xsfc\" (UID: \"b4dc2942-fb5d-473b-93af-c02d761e2d5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:41.168073 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.168047 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z89zj\" (UniqueName: \"kubernetes.io/projected/b4dc2942-fb5d-473b-93af-c02d761e2d5d-kube-api-access-z89zj\") pod \"cert-manager-webhook-6486ffd997-7xsfc\" (UID: \"b4dc2942-fb5d-473b-93af-c02d761e2d5d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:41.176690 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.176670 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd"] Jul 06 19:09:41.178868 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.178854 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.182138 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.182118 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:09:41.182391 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.182375 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:09:41.182478 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.182452 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nhb6z\"" Jul 06 19:09:41.195643 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.195627 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd"] Jul 06 19:09:41.199089 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.199070 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:41.341505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.341480 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7xsfc"] Jul 06 19:09:41.343808 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:09:41.343786 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4dc2942_fb5d_473b_93af_c02d761e2d5d.slice/crio-7d6586e8eeb268d99feb11e7588e67e231b0a2b52474c40cde4c3a3b4778e93b WatchSource:0}: Error finding container 7d6586e8eeb268d99feb11e7588e67e231b0a2b52474c40cde4c3a3b4778e93b: Status 404 returned error can't find the container with id 7d6586e8eeb268d99feb11e7588e67e231b0a2b52474c40cde4c3a3b4778e93b Jul 06 19:09:41.358831 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.358807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.358923 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.358841 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.358923 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.358887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhrmd\" (UniqueName: \"kubernetes.io/projected/8614dfa2-1d51-4c0c-96bd-c544d9333eab-kube-api-access-vhrmd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.459954 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.459893 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhrmd\" (UniqueName: \"kubernetes.io/projected/8614dfa2-1d51-4c0c-96bd-c544d9333eab-kube-api-access-vhrmd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.459954 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.459943 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.460106 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.459975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.460248 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.460233 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.460283 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.460274 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.468414 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.468396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhrmd\" (UniqueName: \"kubernetes.io/projected/8614dfa2-1d51-4c0c-96bd-c544d9333eab-kube-api-access-vhrmd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.487777 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.487760 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:41.609548 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.609520 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd"] Jul 06 19:09:41.611690 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:09:41.611664 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8614dfa2_1d51_4c0c_96bd_c544d9333eab.slice/crio-6d6584f49eeb0a942d7f6f0753fcf77fe8a2a7048d6a708c11f752f590466b91 WatchSource:0}: Error finding container 6d6584f49eeb0a942d7f6f0753fcf77fe8a2a7048d6a708c11f752f590466b91: Status 404 returned error can't find the container with id 6d6584f49eeb0a942d7f6f0753fcf77fe8a2a7048d6a708c11f752f590466b91 Jul 06 19:09:41.875469 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.875437 2569 generic.go:356] "Generic (PLEG): container finished" podID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerID="c9b35a7a886d49f38637dd03ff02b419e53658de340072d9a8612e3ff9fb4093" exitCode=0 Jul 06 19:09:41.875612 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.875515 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" event={"ID":"8614dfa2-1d51-4c0c-96bd-c544d9333eab","Type":"ContainerDied","Data":"c9b35a7a886d49f38637dd03ff02b419e53658de340072d9a8612e3ff9fb4093"} Jul 06 19:09:41.875612 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.875563 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" event={"ID":"8614dfa2-1d51-4c0c-96bd-c544d9333eab","Type":"ContainerStarted","Data":"6d6584f49eeb0a942d7f6f0753fcf77fe8a2a7048d6a708c11f752f590466b91"} Jul 06 19:09:41.876776 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:41.876754 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" event={"ID":"b4dc2942-fb5d-473b-93af-c02d761e2d5d","Type":"ContainerStarted","Data":"7d6586e8eeb268d99feb11e7588e67e231b0a2b52474c40cde4c3a3b4778e93b"} Jul 06 19:09:42.748628 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:42.748596 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:42.971013 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:42.970971 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:44.891649 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:44.891577 2569 generic.go:356] "Generic (PLEG): container finished" podID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerID="fa09844c5a8ee9a6f08f6aed4ee4085833de6453ef009834039d72b077e16182" exitCode=0 Jul 06 19:09:44.892025 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:44.891666 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" event={"ID":"8614dfa2-1d51-4c0c-96bd-c544d9333eab","Type":"ContainerDied","Data":"fa09844c5a8ee9a6f08f6aed4ee4085833de6453ef009834039d72b077e16182"} Jul 06 19:09:44.893283 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:44.893263 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" event={"ID":"b4dc2942-fb5d-473b-93af-c02d761e2d5d","Type":"ContainerStarted","Data":"1714693425d86cfa072aa6029d4501f201cbf2e0f358cea46f4c1622ccd414c1"} Jul 06 19:09:44.893366 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:44.893304 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:44.928051 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:44.928006 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" podStartSLOduration=1.627918372 podStartE2EDuration="4.927993326s" totalImagesPullingTime="3.300074954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:44.926485223 +0000 UTC m=+212.467207454" watchObservedRunningTime="2026-07-06 19:09:44.927993326 +0000 UTC m=+212.468715554" Jul 06 19:09:45.256259 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:45.256179 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:45.899303 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:45.899267 2569 generic.go:356] "Generic (PLEG): container finished" podID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerID="32c9baaccc034eef87fa767b8dd6c8050daab714ecbf43795519807b1f280c90" exitCode=0 Jul 06 19:09:45.899705 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:45.899333 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" event={"ID":"8614dfa2-1d51-4c0c-96bd-c544d9333eab","Type":"ContainerDied","Data":"32c9baaccc034eef87fa767b8dd6c8050daab714ecbf43795519807b1f280c90"} Jul 06 19:09:46.133999 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:46.133967 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:46.198802 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:46.198702 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:47.039634 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.039612 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:47.199652 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.199586 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-bundle\") pod \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " Jul 06 19:09:47.199652 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.199625 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhrmd\" (UniqueName: \"kubernetes.io/projected/8614dfa2-1d51-4c0c-96bd-c544d9333eab-kube-api-access-vhrmd\") pod \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " Jul 06 19:09:47.199811 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.199674 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-util\") pod \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\" (UID: \"8614dfa2-1d51-4c0c-96bd-c544d9333eab\") " Jul 06 19:09:47.200009 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.199922 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-bundle" (OuterVolumeSpecName: "bundle") pod "8614dfa2-1d51-4c0c-96bd-c544d9333eab" (UID: "8614dfa2-1d51-4c0c-96bd-c544d9333eab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:09:47.200208 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.200179 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-bundle\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:09:47.201587 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.201567 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8614dfa2-1d51-4c0c-96bd-c544d9333eab-kube-api-access-vhrmd" (OuterVolumeSpecName: "kube-api-access-vhrmd") pod "8614dfa2-1d51-4c0c-96bd-c544d9333eab" (UID: "8614dfa2-1d51-4c0c-96bd-c544d9333eab"). InnerVolumeSpecName "kube-api-access-vhrmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:09:47.204047 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.204019 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-util" (OuterVolumeSpecName: "util") pod "8614dfa2-1d51-4c0c-96bd-c544d9333eab" (UID: "8614dfa2-1d51-4c0c-96bd-c544d9333eab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:09:47.301052 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.301029 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhrmd\" (UniqueName: \"kubernetes.io/projected/8614dfa2-1d51-4c0c-96bd-c544d9333eab-kube-api-access-vhrmd\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:09:47.301052 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.301051 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8614dfa2-1d51-4c0c-96bd-c544d9333eab-util\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:09:47.908954 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.908919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" event={"ID":"8614dfa2-1d51-4c0c-96bd-c544d9333eab","Type":"ContainerDied","Data":"6d6584f49eeb0a942d7f6f0753fcf77fe8a2a7048d6a708c11f752f590466b91"} Jul 06 19:09:47.908954 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.908936 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fb6hcd" Jul 06 19:09:47.909299 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:47.908964 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d6584f49eeb0a942d7f6f0753fcf77fe8a2a7048d6a708c11f752f590466b91" Jul 06 19:09:48.525925 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.525890 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9pln6"] Jul 06 19:09:48.526287 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.526178 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerName="pull" Jul 06 19:09:48.526287 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.526192 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerName="pull" Jul 06 19:09:48.526287 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.526207 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerName="extract" Jul 06 19:09:48.526287 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.526212 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerName="extract" Jul 06 19:09:48.526287 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.526229 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerName="util" Jul 06 19:09:48.526287 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.526234 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerName="util" Jul 06 19:09:48.526287 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.526282 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8614dfa2-1d51-4c0c-96bd-c544d9333eab" containerName="extract" Jul 06 19:09:48.528487 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.528472 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" Jul 06 19:09:48.531883 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.531866 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gnkfp\"" Jul 06 19:09:48.539119 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.539096 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9pln6"] Jul 06 19:09:48.608189 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.608166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zh4\" (UniqueName: \"kubernetes.io/projected/9d813a1a-ef3c-4e9a-a1fc-c958388f015a-kube-api-access-m4zh4\") pod \"cert-manager-cainjector-559cdd8c87-9pln6\" (UID: \"9d813a1a-ef3c-4e9a-a1fc-c958388f015a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" Jul 06 19:09:48.608281 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.608215 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d813a1a-ef3c-4e9a-a1fc-c958388f015a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9pln6\" (UID: \"9d813a1a-ef3c-4e9a-a1fc-c958388f015a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" Jul 06 19:09:48.709481 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.709458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d813a1a-ef3c-4e9a-a1fc-c958388f015a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9pln6\" (UID: \"9d813a1a-ef3c-4e9a-a1fc-c958388f015a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" Jul 06 19:09:48.709570 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.709504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zh4\" (UniqueName: \"kubernetes.io/projected/9d813a1a-ef3c-4e9a-a1fc-c958388f015a-kube-api-access-m4zh4\") pod \"cert-manager-cainjector-559cdd8c87-9pln6\" (UID: \"9d813a1a-ef3c-4e9a-a1fc-c958388f015a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" Jul 06 19:09:48.719193 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.719164 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d813a1a-ef3c-4e9a-a1fc-c958388f015a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9pln6\" (UID: \"9d813a1a-ef3c-4e9a-a1fc-c958388f015a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" Jul 06 19:09:48.719273 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.719176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zh4\" (UniqueName: \"kubernetes.io/projected/9d813a1a-ef3c-4e9a-a1fc-c958388f015a-kube-api-access-m4zh4\") pod \"cert-manager-cainjector-559cdd8c87-9pln6\" (UID: \"9d813a1a-ef3c-4e9a-a1fc-c958388f015a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" Jul 06 19:09:48.837284 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.837221 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" Jul 06 19:09:48.962131 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:48.962105 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9pln6"] Jul 06 19:09:48.965725 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:09:48.965699 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d813a1a_ef3c_4e9a_a1fc_c958388f015a.slice/crio-026652a17eebca3622904b99c92c6ca861e1e5bf319320966264e254eeb033da WatchSource:0}: Error finding container 026652a17eebca3622904b99c92c6ca861e1e5bf319320966264e254eeb033da: Status 404 returned error can't find the container with id 026652a17eebca3622904b99c92c6ca861e1e5bf319320966264e254eeb033da Jul 06 19:09:49.741581 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:49.741544 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:49.809219 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:49.809180 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:49.918809 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:49.918780 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" event={"ID":"9d813a1a-ef3c-4e9a-a1fc-c958388f015a","Type":"ContainerStarted","Data":"5b054a77bd73a5198c5a29a27036dc3a2bfc8c82d463fca91df49f3a81cb0e67"} Jul 06 19:09:49.918920 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:49.918814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" event={"ID":"9d813a1a-ef3c-4e9a-a1fc-c958388f015a","Type":"ContainerStarted","Data":"026652a17eebca3622904b99c92c6ca861e1e5bf319320966264e254eeb033da"} Jul 06 19:09:49.937279 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:49.937239 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9pln6" podStartSLOduration=1.937225067 podStartE2EDuration="1.937225067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:49.936724743 +0000 UTC m=+217.477446984" watchObservedRunningTime="2026-07-06 19:09:49.937225067 +0000 UTC m=+217.477947294" Jul 06 19:09:50.628615 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:50.628579 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:50.902064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:50.901994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7xsfc" Jul 06 19:09:51.143674 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.143643 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:51.241829 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.241748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:51.622974 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.622940 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hqzpm"] Jul 06 19:09:51.626021 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.626003 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hqzpm" Jul 06 19:09:51.628189 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.628172 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nvjqd\"" Jul 06 19:09:51.629910 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.629891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtzhh\" (UniqueName: \"kubernetes.io/projected/f59c1b32-62ce-42fe-937f-1c901c9f976c-kube-api-access-qtzhh\") pod \"cert-manager-55b4d954d5-hqzpm\" (UID: \"f59c1b32-62ce-42fe-937f-1c901c9f976c\") " pod="cert-manager/cert-manager-55b4d954d5-hqzpm" Jul 06 19:09:51.629971 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.629936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f59c1b32-62ce-42fe-937f-1c901c9f976c-bound-sa-token\") pod \"cert-manager-55b4d954d5-hqzpm\" (UID: \"f59c1b32-62ce-42fe-937f-1c901c9f976c\") " pod="cert-manager/cert-manager-55b4d954d5-hqzpm" Jul 06 19:09:51.636135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.636111 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hqzpm"] Jul 06 19:09:51.730181 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.730158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f59c1b32-62ce-42fe-937f-1c901c9f976c-bound-sa-token\") pod \"cert-manager-55b4d954d5-hqzpm\" (UID: \"f59c1b32-62ce-42fe-937f-1c901c9f976c\") " pod="cert-manager/cert-manager-55b4d954d5-hqzpm" Jul 06 19:09:51.730284 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.730200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzhh\" (UniqueName: \"kubernetes.io/projected/f59c1b32-62ce-42fe-937f-1c901c9f976c-kube-api-access-qtzhh\") pod \"cert-manager-55b4d954d5-hqzpm\" (UID: \"f59c1b32-62ce-42fe-937f-1c901c9f976c\") " pod="cert-manager/cert-manager-55b4d954d5-hqzpm" Jul 06 19:09:51.738534 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.738510 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f59c1b32-62ce-42fe-937f-1c901c9f976c-bound-sa-token\") pod \"cert-manager-55b4d954d5-hqzpm\" (UID: \"f59c1b32-62ce-42fe-937f-1c901c9f976c\") " pod="cert-manager/cert-manager-55b4d954d5-hqzpm" Jul 06 19:09:51.738706 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.738685 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzhh\" (UniqueName: \"kubernetes.io/projected/f59c1b32-62ce-42fe-937f-1c901c9f976c-kube-api-access-qtzhh\") pod \"cert-manager-55b4d954d5-hqzpm\" (UID: \"f59c1b32-62ce-42fe-937f-1c901c9f976c\") " pod="cert-manager/cert-manager-55b4d954d5-hqzpm" Jul 06 19:09:51.936541 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:51.936483 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hqzpm" Jul 06 19:09:52.055971 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:52.055944 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hqzpm"] Jul 06 19:09:52.059290 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:09:52.059262 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf59c1b32_62ce_42fe_937f_1c901c9f976c.slice/crio-3f896a7ca8013d4ddc26863140fae651efbe85caa9858a90e34d4391577274bb WatchSource:0}: Error finding container 3f896a7ca8013d4ddc26863140fae651efbe85caa9858a90e34d4391577274bb: Status 404 returned error can't find the container with id 3f896a7ca8013d4ddc26863140fae651efbe85caa9858a90e34d4391577274bb Jul 06 19:09:52.932772 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:52.932738 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hqzpm" event={"ID":"f59c1b32-62ce-42fe-937f-1c901c9f976c","Type":"ContainerStarted","Data":"8ce6a3df03c3950125ac63db21c45bc4376424101cea93c42b435be0566ce6d1"} Jul 06 19:09:52.932772 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:52.932775 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hqzpm" event={"ID":"f59c1b32-62ce-42fe-937f-1c901c9f976c","Type":"ContainerStarted","Data":"3f896a7ca8013d4ddc26863140fae651efbe85caa9858a90e34d4391577274bb"} Jul 06 19:09:52.949968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:52.949927 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-hqzpm" podStartSLOduration=1.949911294 podStartE2EDuration="1.949911294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:52.948555602 +0000 UTC m=+220.489277823" watchObservedRunningTime="2026-07-06 19:09:52.949911294 +0000 UTC m=+220.490633522" Jul 06 19:09:57.607568 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:57.607532 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:57.710775 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:57.710747 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:59.650200 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.650170 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8"] Jul 06 19:09:59.694762 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.694736 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8"] Jul 06 19:09:59.694910 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.694827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.697544 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.697521 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nhb6z\"" Jul 06 19:09:59.697692 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.697580 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:09:59.697692 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.697630 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:09:59.788363 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.788340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.788511 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.788369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.788511 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.788396 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvpvh\" (UniqueName: \"kubernetes.io/projected/53439cf7-465a-4aa9-8732-b82c87fa3de0-kube-api-access-mvpvh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.839335 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.839308 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:09:59.889613 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.889589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.889725 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.889623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.889725 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.889670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvpvh\" (UniqueName: \"kubernetes.io/projected/53439cf7-465a-4aa9-8732-b82c87fa3de0-kube-api-access-mvpvh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.890002 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.889981 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.890070 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.890025 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:09:59.898090 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:09:59.898068 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvpvh\" (UniqueName: \"kubernetes.io/projected/53439cf7-465a-4aa9-8732-b82c87fa3de0-kube-api-access-mvpvh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:10:00.004006 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:00.003956 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:10:00.150359 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:00.150339 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8"] Jul 06 19:10:00.152210 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:10:00.152177 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53439cf7_465a_4aa9_8732_b82c87fa3de0.slice/crio-c98939a5c2de102051113aeb705f962777945a6da89896daf1307c3c399e30ef WatchSource:0}: Error finding container c98939a5c2de102051113aeb705f962777945a6da89896daf1307c3c399e30ef: Status 404 returned error can't find the container with id c98939a5c2de102051113aeb705f962777945a6da89896daf1307c3c399e30ef Jul 06 19:10:00.967016 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:00.966983 2569 generic.go:356] "Generic (PLEG): container finished" podID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerID="7e69a0939f281a653fe56d4850191d14dfc0c79e45cf1757ea3fcc025cb3d7b8" exitCode=0 Jul 06 19:10:00.967313 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:00.967041 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" event={"ID":"53439cf7-465a-4aa9-8732-b82c87fa3de0","Type":"ContainerDied","Data":"7e69a0939f281a653fe56d4850191d14dfc0c79e45cf1757ea3fcc025cb3d7b8"} Jul 06 19:10:00.967313 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:00.967083 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" event={"ID":"53439cf7-465a-4aa9-8732-b82c87fa3de0","Type":"ContainerStarted","Data":"c98939a5c2de102051113aeb705f962777945a6da89896daf1307c3c399e30ef"} Jul 06 19:10:02.976620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:02.976588 2569 generic.go:356] "Generic (PLEG): container finished" podID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerID="6849845afbe769159d76cfecb2a0fc6a72ec7ebc33e1364cc8af5b7a53b9330e" exitCode=0 Jul 06 19:10:02.976977 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:02.976656 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" event={"ID":"53439cf7-465a-4aa9-8732-b82c87fa3de0","Type":"ContainerDied","Data":"6849845afbe769159d76cfecb2a0fc6a72ec7ebc33e1364cc8af5b7a53b9330e"} Jul 06 19:10:03.982301 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:03.982265 2569 generic.go:356] "Generic (PLEG): container finished" podID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerID="71a296e381266ed2bca4ee6ac943349d113ab77f1c415024b104828f0a87ee00" exitCode=0 Jul 06 19:10:03.982753 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:03.982313 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" event={"ID":"53439cf7-465a-4aa9-8732-b82c87fa3de0","Type":"ContainerDied","Data":"71a296e381266ed2bca4ee6ac943349d113ab77f1c415024b104828f0a87ee00"} Jul 06 19:10:05.109873 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.109850 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:10:05.230656 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.230625 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvpvh\" (UniqueName: \"kubernetes.io/projected/53439cf7-465a-4aa9-8732-b82c87fa3de0-kube-api-access-mvpvh\") pod \"53439cf7-465a-4aa9-8732-b82c87fa3de0\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " Jul 06 19:10:05.230656 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.230662 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-bundle\") pod \"53439cf7-465a-4aa9-8732-b82c87fa3de0\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " Jul 06 19:10:05.230867 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.230717 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-util\") pod \"53439cf7-465a-4aa9-8732-b82c87fa3de0\" (UID: \"53439cf7-465a-4aa9-8732-b82c87fa3de0\") " Jul 06 19:10:05.231480 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.231448 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-bundle" (OuterVolumeSpecName: "bundle") pod "53439cf7-465a-4aa9-8732-b82c87fa3de0" (UID: "53439cf7-465a-4aa9-8732-b82c87fa3de0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:10:05.232710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.232693 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53439cf7-465a-4aa9-8732-b82c87fa3de0-kube-api-access-mvpvh" (OuterVolumeSpecName: "kube-api-access-mvpvh") pod "53439cf7-465a-4aa9-8732-b82c87fa3de0" (UID: "53439cf7-465a-4aa9-8732-b82c87fa3de0"). InnerVolumeSpecName "kube-api-access-mvpvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:10:05.238715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.238690 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-util" (OuterVolumeSpecName: "util") pod "53439cf7-465a-4aa9-8732-b82c87fa3de0" (UID: "53439cf7-465a-4aa9-8732-b82c87fa3de0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:10:05.331686 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.331659 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvpvh\" (UniqueName: \"kubernetes.io/projected/53439cf7-465a-4aa9-8732-b82c87fa3de0-kube-api-access-mvpvh\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:05.331686 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.331681 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-bundle\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:05.331686 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.331691 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53439cf7-465a-4aa9-8732-b82c87fa3de0-util\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:05.991970 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.991943 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" Jul 06 19:10:05.992124 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.991939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9snd8" event={"ID":"53439cf7-465a-4aa9-8732-b82c87fa3de0","Type":"ContainerDied","Data":"c98939a5c2de102051113aeb705f962777945a6da89896daf1307c3c399e30ef"} Jul 06 19:10:05.992124 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:05.992059 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c98939a5c2de102051113aeb705f962777945a6da89896daf1307c3c399e30ef" Jul 06 19:10:16.551904 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:16.551866 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:16.652701 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:16.652675 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:17.186657 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.186631 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:17.484218 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.484138 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:17.577771 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.577734 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:17.605468 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.605440 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m"] Jul 06 19:10:17.605735 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.605723 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerName="pull" Jul 06 19:10:17.605774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.605738 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerName="pull" Jul 06 19:10:17.605774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.605748 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerName="extract" Jul 06 19:10:17.605774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.605753 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerName="extract" Jul 06 19:10:17.605774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.605763 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerName="util" Jul 06 19:10:17.605774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.605768 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerName="util" Jul 06 19:10:17.605913 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.605829 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="53439cf7-465a-4aa9-8732-b82c87fa3de0" containerName="extract" Jul 06 19:10:17.609047 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.609030 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.611447 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.611406 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:10:17.611552 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.611480 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:10:17.611552 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.611488 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:10:17.611745 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.611728 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-59vkn\"" Jul 06 19:10:17.611811 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.611729 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:10:17.622818 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.622801 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m"] Jul 06 19:10:17.719863 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.719835 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgntj\" (UniqueName: \"kubernetes.io/projected/e4d03638-f6f0-4d19-8709-06c9e1903f61-kube-api-access-lgntj\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.719989 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.719879 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4d03638-f6f0-4d19-8709-06c9e1903f61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.719989 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.719904 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4d03638-f6f0-4d19-8709-06c9e1903f61-manifests\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.720058 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.719987 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4d03638-f6f0-4d19-8709-06c9e1903f61-tmp\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.720058 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.720020 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4d03638-f6f0-4d19-8709-06c9e1903f61-webhook-cert\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.758805 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.758752 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:17.821156 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.821135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4d03638-f6f0-4d19-8709-06c9e1903f61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.821233 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.821167 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4d03638-f6f0-4d19-8709-06c9e1903f61-manifests\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.821233 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.821198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4d03638-f6f0-4d19-8709-06c9e1903f61-tmp\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.821233 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.821218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4d03638-f6f0-4d19-8709-06c9e1903f61-webhook-cert\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.821357 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.821238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgntj\" (UniqueName: \"kubernetes.io/projected/e4d03638-f6f0-4d19-8709-06c9e1903f61-kube-api-access-lgntj\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.821609 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.821586 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4d03638-f6f0-4d19-8709-06c9e1903f61-manifests\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.823346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.823319 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4d03638-f6f0-4d19-8709-06c9e1903f61-tmp\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.823614 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.823596 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4d03638-f6f0-4d19-8709-06c9e1903f61-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.823719 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.823698 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4d03638-f6f0-4d19-8709-06c9e1903f61-webhook-cert\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.832483 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.832465 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgntj\" (UniqueName: \"kubernetes.io/projected/e4d03638-f6f0-4d19-8709-06c9e1903f61-kube-api-access-lgntj\") pod \"opendatahub-operator-controller-manager-7598fdd968-9ld5m\" (UID: \"e4d03638-f6f0-4d19-8709-06c9e1903f61\") " pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.859573 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.859553 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:17.919354 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.919335 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:17.960535 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:17.960504 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:18.074247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:18.074213 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:18.104526 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:18.104505 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m"] Jul 06 19:10:18.106694 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:10:18.106673 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d03638_f6f0_4d19_8709_06c9e1903f61.slice/crio-403ade4228d1af4a389a9f426fddea9cd6e3ee9814add07ad380ff443cfe99fc WatchSource:0}: Error finding container 403ade4228d1af4a389a9f426fddea9cd6e3ee9814add07ad380ff443cfe99fc: Status 404 returned error can't find the container with id 403ade4228d1af4a389a9f426fddea9cd6e3ee9814add07ad380ff443cfe99fc Jul 06 19:10:18.184851 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:18.184827 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:18.281298 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:18.281275 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:18.382847 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:18.382793 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:19.046087 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:19.046048 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" event={"ID":"e4d03638-f6f0-4d19-8709-06c9e1903f61","Type":"ContainerStarted","Data":"403ade4228d1af4a389a9f426fddea9cd6e3ee9814add07ad380ff443cfe99fc"} Jul 06 19:10:21.056809 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:21.056779 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" event={"ID":"e4d03638-f6f0-4d19-8709-06c9e1903f61","Type":"ContainerStarted","Data":"9cec07fa03505a4e8b964622d72ca14cf79a0379ec7dbd0733644812fe808aa2"} Jul 06 19:10:22.062244 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:22.062209 2569 generic.go:356] "Generic (PLEG): container finished" podID="e4d03638-f6f0-4d19-8709-06c9e1903f61" containerID="9cec07fa03505a4e8b964622d72ca14cf79a0379ec7dbd0733644812fe808aa2" exitCode=0 Jul 06 19:10:22.062639 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:22.062256 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" event={"ID":"e4d03638-f6f0-4d19-8709-06c9e1903f61","Type":"ContainerDied","Data":"9cec07fa03505a4e8b964622d72ca14cf79a0379ec7dbd0733644812fe808aa2"} Jul 06 19:10:23.067660 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:23.067625 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" event={"ID":"e4d03638-f6f0-4d19-8709-06c9e1903f61","Type":"ContainerStarted","Data":"ae0f479eca080771a84b3b762d875cd3f5ce2afddee2407c239fb3ba446128c8"} Jul 06 19:10:23.068118 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:23.067760 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:23.093756 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:23.093714 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" podStartSLOduration=2.194980229 podStartE2EDuration="6.093700681s" totalImagesPullingTime="2.950842504s" totalInitContainerRuntime="947.877948ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:23.091137324 +0000 UTC m=+250.631859551" watchObservedRunningTime="2026-07-06 19:10:23.093700681 +0000 UTC m=+250.634422909" Jul 06 19:10:29.593181 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:29.593144 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:29.674320 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:29.674295 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:34.074136 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:34.074110 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7598fdd968-9ld5m" Jul 06 19:10:36.270205 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.270126 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:36.385224 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.385189 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:36.756827 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.756751 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:36.942255 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.942223 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw"] Jul 06 19:10:36.944768 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.944745 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:36.947215 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.947194 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:10:36.948241 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.948224 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:10:36.948393 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.948282 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:10:36.948555 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.948532 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hhbgq\"" Jul 06 19:10:36.948651 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.948638 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:10:36.966225 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.966205 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw"] Jul 06 19:10:36.997639 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:36.997614 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:37.064415 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.064390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9vt\" (UniqueName: \"kubernetes.io/projected/aee4644a-c77a-4cbb-9c5b-574c7158ee06-kube-api-access-vr9vt\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.064564 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.064446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee4644a-c77a-4cbb-9c5b-574c7158ee06-tmp\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.064564 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.064501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee4644a-c77a-4cbb-9c5b-574c7158ee06-tls-certs\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.096192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.096170 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bsqdc"] Jul 06 19:10:37.098457 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.098438 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:37.102230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.102211 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 19:10:37.102374 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.102359 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4zvbq\"" Jul 06 19:10:37.116011 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.115979 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bsqdc"] Jul 06 19:10:37.165169 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.165137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9vt\" (UniqueName: \"kubernetes.io/projected/aee4644a-c77a-4cbb-9c5b-574c7158ee06-kube-api-access-vr9vt\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.165355 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.165201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee4644a-c77a-4cbb-9c5b-574c7158ee06-tmp\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.165355 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.165236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rw6d\" (UniqueName: \"kubernetes.io/projected/78ea109b-a990-47ca-a49c-b8737866465c-kube-api-access-5rw6d\") pod \"odh-model-controller-7998c67dc4-bsqdc\" (UID: \"78ea109b-a990-47ca-a49c-b8737866465c\") " pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:37.165355 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.165306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee4644a-c77a-4cbb-9c5b-574c7158ee06-tls-certs\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.165355 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.165351 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78ea109b-a990-47ca-a49c-b8737866465c-cert\") pod \"odh-model-controller-7998c67dc4-bsqdc\" (UID: \"78ea109b-a990-47ca-a49c-b8737866465c\") " pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:37.167628 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.167605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aee4644a-c77a-4cbb-9c5b-574c7158ee06-tmp\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.168717 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.168161 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee4644a-c77a-4cbb-9c5b-574c7158ee06-tls-certs\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.177116 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.177056 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9vt\" (UniqueName: \"kubernetes.io/projected/aee4644a-c77a-4cbb-9c5b-574c7158ee06-kube-api-access-vr9vt\") pod \"kube-auth-proxy-7d54fb4f58-twxjw\" (UID: \"aee4644a-c77a-4cbb-9c5b-574c7158ee06\") " pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.203666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.203638 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:37.256317 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.256289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" Jul 06 19:10:37.266843 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.266819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rw6d\" (UniqueName: \"kubernetes.io/projected/78ea109b-a990-47ca-a49c-b8737866465c-kube-api-access-5rw6d\") pod \"odh-model-controller-7998c67dc4-bsqdc\" (UID: \"78ea109b-a990-47ca-a49c-b8737866465c\") " pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:37.266924 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.266902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78ea109b-a990-47ca-a49c-b8737866465c-cert\") pod \"odh-model-controller-7998c67dc4-bsqdc\" (UID: \"78ea109b-a990-47ca-a49c-b8737866465c\") " pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:37.269633 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.269607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78ea109b-a990-47ca-a49c-b8737866465c-cert\") pod \"odh-model-controller-7998c67dc4-bsqdc\" (UID: \"78ea109b-a990-47ca-a49c-b8737866465c\") " pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:37.281395 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.281373 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rw6d\" (UniqueName: \"kubernetes.io/projected/78ea109b-a990-47ca-a49c-b8737866465c-kube-api-access-5rw6d\") pod \"odh-model-controller-7998c67dc4-bsqdc\" (UID: \"78ea109b-a990-47ca-a49c-b8737866465c\") " pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:37.291618 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.291592 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:37.390489 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.390415 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw"] Jul 06 19:10:37.392714 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:10:37.392680 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee4644a_c77a_4cbb_9c5b_574c7158ee06.slice/crio-ffdb988495af612e2a9f0879bcb9d9f42bb967e252c58e68830251db43cdb548 WatchSource:0}: Error finding container ffdb988495af612e2a9f0879bcb9d9f42bb967e252c58e68830251db43cdb548: Status 404 returned error can't find the container with id ffdb988495af612e2a9f0879bcb9d9f42bb967e252c58e68830251db43cdb548 Jul 06 19:10:37.396290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.396270 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:37.408299 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.408268 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:37.540476 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:37.540299 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bsqdc"] Jul 06 19:10:37.543525 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:10:37.543483 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ea109b_a990_47ca_a49c_b8737866465c.slice/crio-bbac6c7f09a785b57ce07e6f77c2a91e8666fcf3ecf0ca92ec91dac9b124ccdf WatchSource:0}: Error finding container bbac6c7f09a785b57ce07e6f77c2a91e8666fcf3ecf0ca92ec91dac9b124ccdf: Status 404 returned error can't find the container with id bbac6c7f09a785b57ce07e6f77c2a91e8666fcf3ecf0ca92ec91dac9b124ccdf Jul 06 19:10:38.069548 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.069514 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-852z6"] Jul 06 19:10:38.072433 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.072393 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:38.074911 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.074885 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 19:10:38.090112 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.090086 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-852z6"] Jul 06 19:10:38.121517 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.121413 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:38.136837 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.136770 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" event={"ID":"78ea109b-a990-47ca-a49c-b8737866465c","Type":"ContainerStarted","Data":"bbac6c7f09a785b57ce07e6f77c2a91e8666fcf3ecf0ca92ec91dac9b124ccdf"} Jul 06 19:10:38.138142 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.138116 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" event={"ID":"aee4644a-c77a-4cbb-9c5b-574c7158ee06","Type":"ContainerStarted","Data":"ffdb988495af612e2a9f0879bcb9d9f42bb967e252c58e68830251db43cdb548"} Jul 06 19:10:38.175271 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.175219 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0467ef6a-0bb0-4492-a487-2578ca2b9c9d-tls-certs\") pod \"model-serving-api-54d89bd5fc-852z6\" (UID: \"0467ef6a-0bb0-4492-a487-2578ca2b9c9d\") " pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:38.175271 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.175256 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6r6m\" (UniqueName: \"kubernetes.io/projected/0467ef6a-0bb0-4492-a487-2578ca2b9c9d-kube-api-access-r6r6m\") pod \"model-serving-api-54d89bd5fc-852z6\" (UID: \"0467ef6a-0bb0-4492-a487-2578ca2b9c9d\") " pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:38.276061 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.276031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0467ef6a-0bb0-4492-a487-2578ca2b9c9d-tls-certs\") pod \"model-serving-api-54d89bd5fc-852z6\" (UID: \"0467ef6a-0bb0-4492-a487-2578ca2b9c9d\") " pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:38.276228 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.276077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6r6m\" (UniqueName: \"kubernetes.io/projected/0467ef6a-0bb0-4492-a487-2578ca2b9c9d-kube-api-access-r6r6m\") pod \"model-serving-api-54d89bd5fc-852z6\" (UID: \"0467ef6a-0bb0-4492-a487-2578ca2b9c9d\") " pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:38.279800 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.279776 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0467ef6a-0bb0-4492-a487-2578ca2b9c9d-tls-certs\") pod \"model-serving-api-54d89bd5fc-852z6\" (UID: \"0467ef6a-0bb0-4492-a487-2578ca2b9c9d\") " pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:38.295168 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.295114 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6r6m\" (UniqueName: \"kubernetes.io/projected/0467ef6a-0bb0-4492-a487-2578ca2b9c9d-kube-api-access-r6r6m\") pod \"model-serving-api-54d89bd5fc-852z6\" (UID: \"0467ef6a-0bb0-4492-a487-2578ca2b9c9d\") " pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:38.320675 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.320211 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:38.321624 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.320970 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k"] Jul 06 19:10:38.324669 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.324620 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.328890 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.328711 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:10:38.328890 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.328748 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 19:10:38.329043 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.328926 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 19:10:38.329043 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.328975 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 19:10:38.331192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.331170 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nncwp\"" Jul 06 19:10:38.354596 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.354530 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k"] Jul 06 19:10:38.385175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.385041 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:38.455134 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.455097 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:38.477749 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.477714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.477879 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.477779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-local-certs\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.477879 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.477817 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-cacerts\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.477879 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.477840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.478046 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.477884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-token\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.478046 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.477928 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.478046 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.477985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54p4d\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-kube-api-access-54p4d\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.579049 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.578962 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-local-certs\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.579049 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.579012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-cacerts\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.579049 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.579035 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.579304 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.579075 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-token\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.579304 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.579114 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.579304 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.579163 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54p4d\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-kube-api-access-54p4d\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.579304 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.579199 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.580319 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.579804 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.580319 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.580305 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-852z6"] Jul 06 19:10:38.582348 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.582306 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.583238 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.583217 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-cacerts\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.583533 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:10:38.583485 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467ef6a_0bb0_4492_a487_2578ca2b9c9d.slice/crio-394ed0af2ca302d8a4b1138d38a8523cddcd64dcf7430d8d7640e164b2309439 WatchSource:0}: Error finding container 394ed0af2ca302d8a4b1138d38a8523cddcd64dcf7430d8d7640e164b2309439: Status 404 returned error can't find the container with id 394ed0af2ca302d8a4b1138d38a8523cddcd64dcf7430d8d7640e164b2309439 Jul 06 19:10:38.583734 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.583716 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-local-certs\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.584385 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.584170 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.593239 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.590576 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-token\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.596923 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.596895 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54p4d\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-kube-api-access-54p4d\") pod \"istiod-openshift-gateway-7574459656-zsr8k\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.639555 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.639144 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:38.815574 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:38.815197 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k"] Jul 06 19:10:38.817467 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:10:38.817413 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfd3741_c7f0_4008_b075_4a2aebd67c9b.slice/crio-b7f7ffbd36f04fded3bd5e4a9b1cc1682f6e28f49a5588e856932d34450f5c9e WatchSource:0}: Error finding container b7f7ffbd36f04fded3bd5e4a9b1cc1682f6e28f49a5588e856932d34450f5c9e: Status 404 returned error can't find the container with id b7f7ffbd36f04fded3bd5e4a9b1cc1682f6e28f49a5588e856932d34450f5c9e Jul 06 19:10:39.149343 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:39.149301 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" event={"ID":"bbfd3741-c7f0-4008-b075-4a2aebd67c9b","Type":"ContainerStarted","Data":"b7f7ffbd36f04fded3bd5e4a9b1cc1682f6e28f49a5588e856932d34450f5c9e"} Jul 06 19:10:39.151145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:39.151064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-852z6" event={"ID":"0467ef6a-0bb0-4492-a487-2578ca2b9c9d","Type":"ContainerStarted","Data":"394ed0af2ca302d8a4b1138d38a8523cddcd64dcf7430d8d7640e164b2309439"} Jul 06 19:10:39.476611 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:39.476529 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:42.679363 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:42.679331 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:10:42.679651 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:42.679393 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 19:10:43.175626 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.175591 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" event={"ID":"78ea109b-a990-47ca-a49c-b8737866465c","Type":"ContainerStarted","Data":"823c4dbfa08d1516e372c7dcab4d9fd3255bd1010abd024aa906046532849748"} Jul 06 19:10:43.175794 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.175691 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:43.177048 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.177025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" event={"ID":"aee4644a-c77a-4cbb-9c5b-574c7158ee06","Type":"ContainerStarted","Data":"b93192817f51647486d90abe4fb94f912bcad5d70d3f76f0066740929f5e7a45"} Jul 06 19:10:43.178673 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.178649 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" event={"ID":"bbfd3741-c7f0-4008-b075-4a2aebd67c9b","Type":"ContainerStarted","Data":"e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707"} Jul 06 19:10:43.178813 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.178797 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:43.180170 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.180147 2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-zsr8k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:10:43.180256 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.180208 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" podUID="bbfd3741-c7f0-4008-b075-4a2aebd67c9b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:10:43.180398 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.180380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-852z6" event={"ID":"0467ef6a-0bb0-4492-a487-2578ca2b9c9d","Type":"ContainerStarted","Data":"d522df0c4eb83470b8008066b1f31dbbd2385574b95f9e52becd96a812352e5a"} Jul 06 19:10:43.180489 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.180443 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:10:43.197507 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.197456 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" podStartSLOduration=1.119474814 podStartE2EDuration="6.197439102s" totalImagesPullingTime="5.077964288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:43.195641552 +0000 UTC m=+270.736363780" watchObservedRunningTime="2026-07-06 19:10:43.197439102 +0000 UTC m=+270.738161327" Jul 06 19:10:43.218966 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.218918 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" podStartSLOduration=1.359617925 podStartE2EDuration="5.218900374s" totalImagesPullingTime="3.859282449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:43.218697033 +0000 UTC m=+270.759419264" watchObservedRunningTime="2026-07-06 19:10:43.218900374 +0000 UTC m=+270.759622596" Jul 06 19:10:43.235037 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.234994 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d54fb4f58-twxjw" podStartSLOduration=2.006420024 podStartE2EDuration="7.234977425s" totalImagesPullingTime="5.228557401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:43.234380432 +0000 UTC m=+270.775102659" watchObservedRunningTime="2026-07-06 19:10:43.234977425 +0000 UTC m=+270.775699655" Jul 06 19:10:43.257111 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.257068 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-852z6" podStartSLOduration=1.220034521 podStartE2EDuration="5.257042462s" totalImagesPullingTime="4.037007941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:43.254940788 +0000 UTC m=+270.795663016" watchObservedRunningTime="2026-07-06 19:10:43.257042462 +0000 UTC m=+270.797764692" Jul 06 19:10:43.426276 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.426170 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:43.527325 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.527288 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:43.723099 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.723017 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:43.901456 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.901372 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:43.993042 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:43.992942 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:44.185895 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:44.185871 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:48.723731 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.723697 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:48.820385 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.820355 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5"] Jul 06 19:10:48.823405 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.823386 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:48.826979 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.826958 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-spxdw\"" Jul 06 19:10:48.828346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.828330 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 19:10:48.836696 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.836677 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7"] Jul 06 19:10:48.838828 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.838811 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:48.840998 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.840978 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 19:10:48.841096 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.841037 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cntqk\"" Jul 06 19:10:48.842417 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.842390 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5"] Jul 06 19:10:48.859088 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.859065 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7"] Jul 06 19:10:48.976545 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.976486 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54fc970a-68fd-45d1-a054-083b681437ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-br7h7\" (UID: \"54fc970a-68fd-45d1-a054-083b681437ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:48.976545 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.976532 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgkf\" (UniqueName: \"kubernetes.io/projected/54fc970a-68fd-45d1-a054-083b681437ea-kube-api-access-rrgkf\") pod \"llmisvc-controller-manager-777c6bd7d5-br7h7\" (UID: \"54fc970a-68fd-45d1-a054-083b681437ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:48.976696 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.976563 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbkmf\" (UniqueName: \"kubernetes.io/projected/8f7497cb-d60d-48d7-9e8d-b219a58455b5-kube-api-access-vbkmf\") pod \"kserve-controller-manager-d6f4b65b8-z4lt5\" (UID: \"8f7497cb-d60d-48d7-9e8d-b219a58455b5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:48.976696 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:48.976610 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7497cb-d60d-48d7-9e8d-b219a58455b5-cert\") pod \"kserve-controller-manager-d6f4b65b8-z4lt5\" (UID: \"8f7497cb-d60d-48d7-9e8d-b219a58455b5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:49.077499 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.077466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54fc970a-68fd-45d1-a054-083b681437ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-br7h7\" (UID: \"54fc970a-68fd-45d1-a054-083b681437ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:49.077608 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.077515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgkf\" (UniqueName: \"kubernetes.io/projected/54fc970a-68fd-45d1-a054-083b681437ea-kube-api-access-rrgkf\") pod \"llmisvc-controller-manager-777c6bd7d5-br7h7\" (UID: \"54fc970a-68fd-45d1-a054-083b681437ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:49.077608 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.077532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbkmf\" (UniqueName: \"kubernetes.io/projected/8f7497cb-d60d-48d7-9e8d-b219a58455b5-kube-api-access-vbkmf\") pod \"kserve-controller-manager-d6f4b65b8-z4lt5\" (UID: \"8f7497cb-d60d-48d7-9e8d-b219a58455b5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:49.077608 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.077558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7497cb-d60d-48d7-9e8d-b219a58455b5-cert\") pod \"kserve-controller-manager-d6f4b65b8-z4lt5\" (UID: \"8f7497cb-d60d-48d7-9e8d-b219a58455b5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:49.077720 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:10:49.077606 2569 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 19:10:49.077720 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:10:49.077649 2569 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 19:10:49.077720 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:10:49.077685 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fc970a-68fd-45d1-a054-083b681437ea-cert podName:54fc970a-68fd-45d1-a054-083b681437ea nodeName:}" failed. No retries permitted until 2026-07-06 19:10:49.577666639 +0000 UTC m=+277.118388852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/54fc970a-68fd-45d1-a054-083b681437ea-cert") pod "llmisvc-controller-manager-777c6bd7d5-br7h7" (UID: "54fc970a-68fd-45d1-a054-083b681437ea") : secret "llmisvc-webhook-server-cert" not found Jul 06 19:10:49.077720 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:10:49.077704 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f7497cb-d60d-48d7-9e8d-b219a58455b5-cert podName:8f7497cb-d60d-48d7-9e8d-b219a58455b5 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:49.577695154 +0000 UTC m=+277.118417361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8f7497cb-d60d-48d7-9e8d-b219a58455b5-cert") pod "kserve-controller-manager-d6f4b65b8-z4lt5" (UID: "8f7497cb-d60d-48d7-9e8d-b219a58455b5") : secret "kserve-webhook-server-cert" not found Jul 06 19:10:49.089605 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.089577 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgkf\" (UniqueName: \"kubernetes.io/projected/54fc970a-68fd-45d1-a054-083b681437ea-kube-api-access-rrgkf\") pod \"llmisvc-controller-manager-777c6bd7d5-br7h7\" (UID: \"54fc970a-68fd-45d1-a054-083b681437ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:49.090770 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.090752 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbkmf\" (UniqueName: \"kubernetes.io/projected/8f7497cb-d60d-48d7-9e8d-b219a58455b5-kube-api-access-vbkmf\") pod \"kserve-controller-manager-d6f4b65b8-z4lt5\" (UID: \"8f7497cb-d60d-48d7-9e8d-b219a58455b5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:49.582102 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.582061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54fc970a-68fd-45d1-a054-083b681437ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-br7h7\" (UID: \"54fc970a-68fd-45d1-a054-083b681437ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:49.582283 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.582128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7497cb-d60d-48d7-9e8d-b219a58455b5-cert\") pod \"kserve-controller-manager-d6f4b65b8-z4lt5\" (UID: \"8f7497cb-d60d-48d7-9e8d-b219a58455b5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:49.584639 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.584609 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54fc970a-68fd-45d1-a054-083b681437ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-br7h7\" (UID: \"54fc970a-68fd-45d1-a054-083b681437ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:49.584881 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.584857 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f7497cb-d60d-48d7-9e8d-b219a58455b5-cert\") pod \"kserve-controller-manager-d6f4b65b8-z4lt5\" (UID: \"8f7497cb-d60d-48d7-9e8d-b219a58455b5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:49.733200 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.733178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:49.749265 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.749247 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:49.862463 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.862435 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5"] Jul 06 19:10:49.864628 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:10:49.864604 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7497cb_d60d_48d7_9e8d_b219a58455b5.slice/crio-15d157c348eef3c91501e4fd3cc7e1b066c73b0b5c959db6a0b0f5cd74c2c421 WatchSource:0}: Error finding container 15d157c348eef3c91501e4fd3cc7e1b066c73b0b5c959db6a0b0f5cd74c2c421: Status 404 returned error can't find the container with id 15d157c348eef3c91501e4fd3cc7e1b066c73b0b5c959db6a0b0f5cd74c2c421 Jul 06 19:10:49.887520 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:49.887499 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7"] Jul 06 19:10:49.890209 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:10:49.890188 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod54fc970a_68fd_45d1_a054_083b681437ea.slice/crio-38dd5245dd1ffd64fde2b6edfa6fd043e30414645e888462e4aa41dd3fc5edfe WatchSource:0}: Error finding container 38dd5245dd1ffd64fde2b6edfa6fd043e30414645e888462e4aa41dd3fc5edfe: Status 404 returned error can't find the container with id 38dd5245dd1ffd64fde2b6edfa6fd043e30414645e888462e4aa41dd3fc5edfe Jul 06 19:10:50.178323 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:50.178262 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:50.213412 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:50.213381 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" event={"ID":"54fc970a-68fd-45d1-a054-083b681437ea","Type":"ContainerStarted","Data":"38dd5245dd1ffd64fde2b6edfa6fd043e30414645e888462e4aa41dd3fc5edfe"} Jul 06 19:10:50.214536 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:50.214515 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" event={"ID":"8f7497cb-d60d-48d7-9e8d-b219a58455b5","Type":"ContainerStarted","Data":"15d157c348eef3c91501e4fd3cc7e1b066c73b0b5c959db6a0b0f5cd74c2c421"} Jul 06 19:10:50.287099 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:50.287069 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:10:51.240440 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:51.240392 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k"] Jul 06 19:10:51.240868 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:51.240631 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" podUID="bbfd3741-c7f0-4008-b075-4a2aebd67c9b" containerName="discovery" containerID="cri-o://e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707" gracePeriod=30 Jul 06 19:10:51.999902 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:51.999878 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:52.003631 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.003607 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-token\") pod \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " Jul 06 19:10:52.003757 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.003651 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-dns-cert\") pod \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " Jul 06 19:10:52.003757 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.003680 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-ca-configmap\") pod \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " Jul 06 19:10:52.003757 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.003734 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-cacerts\") pod \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " Jul 06 19:10:52.003910 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.003793 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54p4d\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-kube-api-access-54p4d\") pod \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " Jul 06 19:10:52.003910 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.003818 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-local-certs\") pod \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " Jul 06 19:10:52.003910 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.003843 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-kubeconfig\") pod \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\" (UID: \"bbfd3741-c7f0-4008-b075-4a2aebd67c9b\") " Jul 06 19:10:52.004084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.004054 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "bbfd3741-c7f0-4008-b075-4a2aebd67c9b" (UID: "bbfd3741-c7f0-4008-b075-4a2aebd67c9b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:10:52.006201 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.006176 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "bbfd3741-c7f0-4008-b075-4a2aebd67c9b" (UID: "bbfd3741-c7f0-4008-b075-4a2aebd67c9b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:10:52.006303 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.006195 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-cacerts" (OuterVolumeSpecName: "cacerts") pod "bbfd3741-c7f0-4008-b075-4a2aebd67c9b" (UID: "bbfd3741-c7f0-4008-b075-4a2aebd67c9b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:10:52.006303 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.006246 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-kube-api-access-54p4d" (OuterVolumeSpecName: "kube-api-access-54p4d") pod "bbfd3741-c7f0-4008-b075-4a2aebd67c9b" (UID: "bbfd3741-c7f0-4008-b075-4a2aebd67c9b"). InnerVolumeSpecName "kube-api-access-54p4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:10:52.006561 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.006528 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbfd3741-c7f0-4008-b075-4a2aebd67c9b" (UID: "bbfd3741-c7f0-4008-b075-4a2aebd67c9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:10:52.006994 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.006966 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "bbfd3741-c7f0-4008-b075-4a2aebd67c9b" (UID: "bbfd3741-c7f0-4008-b075-4a2aebd67c9b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:10:52.007159 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.007134 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-local-certs" (OuterVolumeSpecName: "local-certs") pod "bbfd3741-c7f0-4008-b075-4a2aebd67c9b" (UID: "bbfd3741-c7f0-4008-b075-4a2aebd67c9b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:10:52.104752 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.104717 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-token\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:52.104752 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.104753 2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-dns-cert\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:52.104980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.104767 2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-csr-ca-configmap\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:52.104980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.104779 2569 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-cacerts\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:52.104980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.104793 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54p4d\" (UniqueName: \"kubernetes.io/projected/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-kube-api-access-54p4d\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:52.104980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.104807 2569 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-local-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:52.104980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.104820 2569 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbfd3741-c7f0-4008-b075-4a2aebd67c9b-istio-kubeconfig\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:10:52.227410 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.227376 2569 generic.go:356] "Generic (PLEG): container finished" podID="bbfd3741-c7f0-4008-b075-4a2aebd67c9b" containerID="e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707" exitCode=0 Jul 06 19:10:52.227590 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.227420 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" Jul 06 19:10:52.227590 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.227461 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" event={"ID":"bbfd3741-c7f0-4008-b075-4a2aebd67c9b","Type":"ContainerDied","Data":"e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707"} Jul 06 19:10:52.227590 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.227516 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k" event={"ID":"bbfd3741-c7f0-4008-b075-4a2aebd67c9b","Type":"ContainerDied","Data":"b7f7ffbd36f04fded3bd5e4a9b1cc1682f6e28f49a5588e856932d34450f5c9e"} Jul 06 19:10:52.227590 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.227541 2569 scope.go:117] "RemoveContainer" containerID="e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707" Jul 06 19:10:52.259932 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.259906 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k"] Jul 06 19:10:52.265155 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.265128 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-zsr8k"] Jul 06 19:10:52.538544 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.538476 2569 scope.go:117] "RemoveContainer" containerID="e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707" Jul 06 19:10:52.538827 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:10:52.538804 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707\": container with ID starting with e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707 not found: ID does not exist" containerID="e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707" Jul 06 19:10:52.538903 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:52.538840 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707"} err="failed to get container status \"e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707\": rpc error: code = NotFound desc = could not find container \"e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707\": container with ID starting with e36c99bda6e1db677ab9fcf49291c4228712100b0842d0bffc11cbd74a286707 not found: ID does not exist" Jul 06 19:10:53.020877 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:53.020847 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbfd3741-c7f0-4008-b075-4a2aebd67c9b" path="/var/lib/kubelet/pods/bbfd3741-c7f0-4008-b075-4a2aebd67c9b/volumes" Jul 06 19:10:53.232847 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:53.232804 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" event={"ID":"54fc970a-68fd-45d1-a054-083b681437ea","Type":"ContainerStarted","Data":"dd5c770d09e85fc96e6ad71bb3ea87270cf1139251428f3a7dfbd73ef3a4659f"} Jul 06 19:10:53.233019 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:53.233000 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:10:53.235114 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:53.235092 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" event={"ID":"8f7497cb-d60d-48d7-9e8d-b219a58455b5","Type":"ContainerStarted","Data":"e808e76e5da8cb6a8f152cf0f3f17938d51fc714a3ab0eb3b00471a4ac8277b1"} Jul 06 19:10:53.235201 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:53.235144 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:10:53.284009 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:53.283938 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" podStartSLOduration=2.628956375 podStartE2EDuration="5.283926732s" totalImagesPullingTime="2.654970357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:53.283699978 +0000 UTC m=+280.824422219" watchObservedRunningTime="2026-07-06 19:10:53.283926732 +0000 UTC m=+280.824649004" Jul 06 19:10:53.322354 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:53.322315 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" podStartSLOduration=2.639722318 podStartE2EDuration="5.322301511s" totalImagesPullingTime="2.682579193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:53.322213393 +0000 UTC m=+280.862935624" watchObservedRunningTime="2026-07-06 19:10:53.322301511 +0000 UTC m=+280.863023739" Jul 06 19:10:54.187065 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:54.187033 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-bsqdc" Jul 06 19:10:54.189979 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:10:54.189959 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-852z6" Jul 06 19:11:12.848370 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:12.848347 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:11:12.850955 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:12.850936 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:11:12.855181 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:12.855163 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:11:24.242908 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:24.242880 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-br7h7" Jul 06 19:11:24.251504 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:24.245828 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z4lt5" Jul 06 19:11:35.320531 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:35.320491 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:11:35.385769 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:35.385740 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:11:37.056532 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:37.056487 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:11:37.863758 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:37.863726 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:11:38.073181 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.073149 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2k5zq"] Jul 06 19:11:38.073576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.073489 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfd3741-c7f0-4008-b075-4a2aebd67c9b" containerName="discovery" Jul 06 19:11:38.073576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.073509 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfd3741-c7f0-4008-b075-4a2aebd67c9b" containerName="discovery" Jul 06 19:11:38.073695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.073586 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfd3741-c7f0-4008-b075-4a2aebd67c9b" containerName="discovery" Jul 06 19:11:38.075648 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.075633 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" Jul 06 19:11:38.088870 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.088850 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7ckl2\"" Jul 06 19:11:38.088870 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.088862 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:11:38.089066 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.088905 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:11:38.110230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.110208 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2k5zq"] Jul 06 19:11:38.126290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.126239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czw8l\" (UniqueName: \"kubernetes.io/projected/db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6-kube-api-access-czw8l\") pod \"kuadrant-operator-catalog-2k5zq\" (UID: \"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6\") " pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" Jul 06 19:11:38.135757 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.135734 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:11:38.226791 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.226769 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czw8l\" (UniqueName: \"kubernetes.io/projected/db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6-kube-api-access-czw8l\") pod \"kuadrant-operator-catalog-2k5zq\" (UID: \"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6\") " pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" Jul 06 19:11:38.238925 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.238906 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czw8l\" (UniqueName: \"kubernetes.io/projected/db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6-kube-api-access-czw8l\") pod \"kuadrant-operator-catalog-2k5zq\" (UID: \"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6\") " pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" Jul 06 19:11:38.241726 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.241707 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:11:38.320154 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.320130 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2k5zq"] Jul 06 19:11:38.320345 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.320335 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" Jul 06 19:11:38.460951 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.460932 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2k5zq"] Jul 06 19:11:38.462577 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:11:38.462551 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8d3df6_fbe8_4b92_9be6_4fa9f4ac7fe6.slice/crio-dd8e350b51475c4715f2fa24c80d8a905d7aad8aaeed212c1bb979f7d7dca3d4 WatchSource:0}: Error finding container dd8e350b51475c4715f2fa24c80d8a905d7aad8aaeed212c1bb979f7d7dca3d4: Status 404 returned error can't find the container with id dd8e350b51475c4715f2fa24c80d8a905d7aad8aaeed212c1bb979f7d7dca3d4 Jul 06 19:11:38.463779 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.463764 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:11:38.550451 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.549250 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lkmfh"] Jul 06 19:11:38.553765 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.553745 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:38.566845 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.566827 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lkmfh"] Jul 06 19:11:38.630095 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.630073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7xz\" (UniqueName: \"kubernetes.io/projected/2fbefc5e-5509-46c2-ab17-fb464a89bc65-kube-api-access-hz7xz\") pod \"kuadrant-operator-catalog-lkmfh\" (UID: \"2fbefc5e-5509-46c2-ab17-fb464a89bc65\") " pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:38.731400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.731329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7xz\" (UniqueName: \"kubernetes.io/projected/2fbefc5e-5509-46c2-ab17-fb464a89bc65-kube-api-access-hz7xz\") pod \"kuadrant-operator-catalog-lkmfh\" (UID: \"2fbefc5e-5509-46c2-ab17-fb464a89bc65\") " pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:38.743733 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.743708 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7xz\" (UniqueName: \"kubernetes.io/projected/2fbefc5e-5509-46c2-ab17-fb464a89bc65-kube-api-access-hz7xz\") pod \"kuadrant-operator-catalog-lkmfh\" (UID: \"2fbefc5e-5509-46c2-ab17-fb464a89bc65\") " pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:38.862645 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.862614 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:38.994575 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:38.994460 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lkmfh"] Jul 06 19:11:39.170894 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:11:39.170859 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbefc5e_5509_46c2_ab17_fb464a89bc65.slice/crio-63f310dbd1a957daad87b7bf34aa54c84ab0669ea53b81ec5b79ef2ea1d7bc4f WatchSource:0}: Error finding container 63f310dbd1a957daad87b7bf34aa54c84ab0669ea53b81ec5b79ef2ea1d7bc4f: Status 404 returned error can't find the container with id 63f310dbd1a957daad87b7bf34aa54c84ab0669ea53b81ec5b79ef2ea1d7bc4f Jul 06 19:11:39.449370 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:39.449336 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" event={"ID":"2fbefc5e-5509-46c2-ab17-fb464a89bc65","Type":"ContainerStarted","Data":"63f310dbd1a957daad87b7bf34aa54c84ab0669ea53b81ec5b79ef2ea1d7bc4f"} Jul 06 19:11:39.450621 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:39.450591 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" event={"ID":"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6","Type":"ContainerStarted","Data":"dd8e350b51475c4715f2fa24c80d8a905d7aad8aaeed212c1bb979f7d7dca3d4"} Jul 06 19:11:40.456572 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.456480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" event={"ID":"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6","Type":"ContainerStarted","Data":"fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48"} Jul 06 19:11:40.456572 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.456538 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" podUID="db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6" containerName="registry-server" containerID="cri-o://fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48" gracePeriod=2 Jul 06 19:11:40.458128 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.458098 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" event={"ID":"2fbefc5e-5509-46c2-ab17-fb464a89bc65","Type":"ContainerStarted","Data":"9b7d50369ede9569181be29c64dd39b2486f9ef90f3bd5b78493c137695fb1d5"} Jul 06 19:11:40.474376 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.474335 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" podStartSLOduration=0.815933499 podStartE2EDuration="2.474321778s" totalImagesPullingTime="1.658388279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:40.473370997 +0000 UTC m=+328.014093226" watchObservedRunningTime="2026-07-06 19:11:40.474321778 +0000 UTC m=+328.015044005" Jul 06 19:11:40.489775 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.489742 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" podStartSLOduration=1.539782641 podStartE2EDuration="2.489730155s" totalImagesPullingTime="949.947514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:40.48888785 +0000 UTC m=+328.029610079" watchObservedRunningTime="2026-07-06 19:11:40.489730155 +0000 UTC m=+328.030452374" Jul 06 19:11:40.690417 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.690395 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" Jul 06 19:11:40.749113 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.749053 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czw8l\" (UniqueName: \"kubernetes.io/projected/db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6-kube-api-access-czw8l\") pod \"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6\" (UID: \"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6\") " Jul 06 19:11:40.751023 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.751005 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6-kube-api-access-czw8l" (OuterVolumeSpecName: "kube-api-access-czw8l") pod "db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6" (UID: "db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6"). InnerVolumeSpecName "kube-api-access-czw8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:40.850380 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:40.850344 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czw8l\" (UniqueName: \"kubernetes.io/projected/db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6-kube-api-access-czw8l\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:41.464710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.464623 2569 generic.go:356] "Generic (PLEG): container finished" podID="db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6" containerID="fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48" exitCode=0 Jul 06 19:11:41.464710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.464674 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" Jul 06 19:11:41.465138 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.464705 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" event={"ID":"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6","Type":"ContainerDied","Data":"fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48"} Jul 06 19:11:41.465138 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.464773 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2k5zq" event={"ID":"db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6","Type":"ContainerDied","Data":"dd8e350b51475c4715f2fa24c80d8a905d7aad8aaeed212c1bb979f7d7dca3d4"} Jul 06 19:11:41.465138 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.464809 2569 scope.go:117] "RemoveContainer" containerID="fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48" Jul 06 19:11:41.473998 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.473982 2569 scope.go:117] "RemoveContainer" containerID="fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48" Jul 06 19:11:41.474257 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:11:41.474238 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48\": container with ID starting with fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48 not found: ID does not exist" containerID="fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48" Jul 06 19:11:41.474301 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.474264 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48"} err="failed to get container status \"fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48\": rpc error: code = NotFound desc = could not find container \"fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48\": container with ID starting with fca98e619bd26cabdf223df97fd44aa62b4d62db0251350348973f9f6aee8d48 not found: ID does not exist" Jul 06 19:11:41.482235 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.482208 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2k5zq"] Jul 06 19:11:41.485802 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:41.485778 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2k5zq"] Jul 06 19:11:43.020748 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:43.020719 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6" path="/var/lib/kubelet/pods/db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6/volumes" Jul 06 19:11:48.863138 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:48.863110 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:48.863610 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:48.863257 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:48.885389 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:48.885366 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:49.526736 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:49.526712 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-lkmfh" Jul 06 19:11:53.399615 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.399586 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z"] Jul 06 19:11:53.399963 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.399929 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6" containerName="registry-server" Jul 06 19:11:53.399963 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.399940 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6" containerName="registry-server" Jul 06 19:11:53.400030 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.400000 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="db8d3df6-fbe8-4b92-9be6-4fa9f4ac7fe6" containerName="registry-server" Jul 06 19:11:53.403157 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.403138 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.405589 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.405572 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n82z9\"" Jul 06 19:11:53.412147 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.412126 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z"] Jul 06 19:11:53.546179 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.546160 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.546292 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.546232 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68b6q\" (UniqueName: \"kubernetes.io/projected/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-kube-api-access-68b6q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.546292 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.546259 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.647544 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.647524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68b6q\" (UniqueName: \"kubernetes.io/projected/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-kube-api-access-68b6q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.647638 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.647551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.647638 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.647580 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.647872 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.647857 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.647933 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.647916 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.656366 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.656314 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68b6q\" (UniqueName: \"kubernetes.io/projected/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-kube-api-access-68b6q\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.711815 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.711797 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:53.851540 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.851519 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z"] Jul 06 19:11:53.853494 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:11:53.853465 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6da782_ce13_47a7_81ad_e084cb1b9ccb.slice/crio-fcee4e85d1646caabbcc1df730babd0cd9cd81f29bb92d35e9439c07b4425078 WatchSource:0}: Error finding container fcee4e85d1646caabbcc1df730babd0cd9cd81f29bb92d35e9439c07b4425078: Status 404 returned error can't find the container with id fcee4e85d1646caabbcc1df730babd0cd9cd81f29bb92d35e9439c07b4425078 Jul 06 19:11:53.906349 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.906330 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q"] Jul 06 19:11:53.909371 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.909353 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:53.930772 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:53.928329 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q"] Jul 06 19:11:54.050988 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.050961 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.051081 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.051021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.051081 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.051047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6l8r\" (UniqueName: \"kubernetes.io/projected/08c50fa4-be66-43d0-8610-19a7e9111fcc-kube-api-access-s6l8r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.152052 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.152025 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.152135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.152098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.152135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.152125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6l8r\" (UniqueName: \"kubernetes.io/projected/08c50fa4-be66-43d0-8610-19a7e9111fcc-kube-api-access-s6l8r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.152528 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.152500 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.152611 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.152510 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.160258 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.160219 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6l8r\" (UniqueName: \"kubernetes.io/projected/08c50fa4-be66-43d0-8610-19a7e9111fcc-kube-api-access-s6l8r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.217583 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.217561 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:54.338686 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.338664 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q"] Jul 06 19:11:54.339693 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:11:54.339672 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c50fa4_be66_43d0_8610_19a7e9111fcc.slice/crio-c8c4ef40a5a0a9781b058f7bdc51c97459bca75fd88088b1d2b1d1a77ce82985 WatchSource:0}: Error finding container c8c4ef40a5a0a9781b058f7bdc51c97459bca75fd88088b1d2b1d1a77ce82985: Status 404 returned error can't find the container with id c8c4ef40a5a0a9781b058f7bdc51c97459bca75fd88088b1d2b1d1a77ce82985 Jul 06 19:11:54.507589 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.507569 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m"] Jul 06 19:11:54.509923 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.509908 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.519255 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.519233 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m"] Jul 06 19:11:54.530610 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.530588 2569 generic.go:356] "Generic (PLEG): container finished" podID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerID="6949c6f09768da3d54ea7be6ff7e1488e253c0624f6a2aaf098838266effb286" exitCode=0 Jul 06 19:11:54.530699 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.530623 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" event={"ID":"08c50fa4-be66-43d0-8610-19a7e9111fcc","Type":"ContainerDied","Data":"6949c6f09768da3d54ea7be6ff7e1488e253c0624f6a2aaf098838266effb286"} Jul 06 19:11:54.530699 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.530662 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" event={"ID":"08c50fa4-be66-43d0-8610-19a7e9111fcc","Type":"ContainerStarted","Data":"c8c4ef40a5a0a9781b058f7bdc51c97459bca75fd88088b1d2b1d1a77ce82985"} Jul 06 19:11:54.532159 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.532139 2569 generic.go:356] "Generic (PLEG): container finished" podID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerID="81933171e98e87c4278ac67f5a72b6955182003629e09004854e2bc1d996006d" exitCode=0 Jul 06 19:11:54.532251 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.532225 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" event={"ID":"8a6da782-ce13-47a7-81ad-e084cb1b9ccb","Type":"ContainerDied","Data":"81933171e98e87c4278ac67f5a72b6955182003629e09004854e2bc1d996006d"} Jul 06 19:11:54.532306 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.532259 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" event={"ID":"8a6da782-ce13-47a7-81ad-e084cb1b9ccb","Type":"ContainerStarted","Data":"fcee4e85d1646caabbcc1df730babd0cd9cd81f29bb92d35e9439c07b4425078"} Jul 06 19:11:54.656084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.656057 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.656182 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.656128 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.656182 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.656152 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhls\" (UniqueName: \"kubernetes.io/projected/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-kube-api-access-nlhls\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.757031 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.756983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.757031 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.757010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhls\" (UniqueName: \"kubernetes.io/projected/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-kube-api-access-nlhls\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.757161 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.757144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.757359 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.757341 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.757445 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.757410 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.765303 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.765278 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhls\" (UniqueName: \"kubernetes.io/projected/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-kube-api-access-nlhls\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.818465 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.818446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:54.919161 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.919131 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22"] Jul 06 19:11:54.922743 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.922722 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:54.932621 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.932599 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22"] Jul 06 19:11:54.945564 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:54.945522 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m"] Jul 06 19:11:54.947797 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:11:54.947775 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933ceb0d_d79f_4e6f_bcd8_4659996a35ec.slice/crio-a3add14d279f598b73185153a2d746033875b4b9035fb53aa434aa0f9ad5a427 WatchSource:0}: Error finding container a3add14d279f598b73185153a2d746033875b4b9035fb53aa434aa0f9ad5a427: Status 404 returned error can't find the container with id a3add14d279f598b73185153a2d746033875b4b9035fb53aa434aa0f9ad5a427 Jul 06 19:11:55.060525 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.060500 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.060657 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.060534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxdr\" (UniqueName: \"kubernetes.io/projected/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-kube-api-access-6gxdr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.060720 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.060651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.161238 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.161213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.161345 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.161248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxdr\" (UniqueName: \"kubernetes.io/projected/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-kube-api-access-6gxdr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.161345 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.161322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.161650 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.161629 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.161695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.161655 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.170163 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.170138 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxdr\" (UniqueName: \"kubernetes.io/projected/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-kube-api-access-6gxdr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.233143 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.233124 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:55.375239 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.375216 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22"] Jul 06 19:11:55.381450 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:11:55.381406 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b6ce61_ff82_4666_80fd_2f55a0ffecbc.slice/crio-54110f07a087f9c5708a59595da5736aeb37e32d13a8387072930556da3c6bcb WatchSource:0}: Error finding container 54110f07a087f9c5708a59595da5736aeb37e32d13a8387072930556da3c6bcb: Status 404 returned error can't find the container with id 54110f07a087f9c5708a59595da5736aeb37e32d13a8387072930556da3c6bcb Jul 06 19:11:55.538763 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.538712 2569 generic.go:356] "Generic (PLEG): container finished" podID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerID="a8f1f65c514a6e4ac8691876efc6ad78d909801d1b285a4e0b267d082daffb82" exitCode=0 Jul 06 19:11:55.539095 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.538801 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" event={"ID":"933ceb0d-d79f-4e6f-bcd8-4659996a35ec","Type":"ContainerDied","Data":"a8f1f65c514a6e4ac8691876efc6ad78d909801d1b285a4e0b267d082daffb82"} Jul 06 19:11:55.539095 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.538860 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" event={"ID":"933ceb0d-d79f-4e6f-bcd8-4659996a35ec","Type":"ContainerStarted","Data":"a3add14d279f598b73185153a2d746033875b4b9035fb53aa434aa0f9ad5a427"} Jul 06 19:11:55.540718 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.540698 2569 generic.go:356] "Generic (PLEG): container finished" podID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerID="20fd137f4c09e4de0f1a0997346313ac0a4127704d5b0a3e22b919648c18eb3c" exitCode=0 Jul 06 19:11:55.540812 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.540789 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" event={"ID":"8a6da782-ce13-47a7-81ad-e084cb1b9ccb","Type":"ContainerDied","Data":"20fd137f4c09e4de0f1a0997346313ac0a4127704d5b0a3e22b919648c18eb3c"} Jul 06 19:11:55.542718 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.542699 2569 generic.go:356] "Generic (PLEG): container finished" podID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerID="635a2e8185b1799ab0daaacf122b096107821b6dac965390f9948a081ab75026" exitCode=0 Jul 06 19:11:55.542810 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.542778 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" event={"ID":"08c50fa4-be66-43d0-8610-19a7e9111fcc","Type":"ContainerDied","Data":"635a2e8185b1799ab0daaacf122b096107821b6dac965390f9948a081ab75026"} Jul 06 19:11:55.544296 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.544276 2569 generic.go:356] "Generic (PLEG): container finished" podID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerID="3ca97d6327e044e320adde419c8bd777cda73e0e66dc0fca636e6fcd8016045c" exitCode=0 Jul 06 19:11:55.544390 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.544309 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" event={"ID":"62b6ce61-ff82-4666-80fd-2f55a0ffecbc","Type":"ContainerDied","Data":"3ca97d6327e044e320adde419c8bd777cda73e0e66dc0fca636e6fcd8016045c"} Jul 06 19:11:55.544390 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:55.544344 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" event={"ID":"62b6ce61-ff82-4666-80fd-2f55a0ffecbc","Type":"ContainerStarted","Data":"54110f07a087f9c5708a59595da5736aeb37e32d13a8387072930556da3c6bcb"} Jul 06 19:11:56.551440 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:56.551400 2569 generic.go:356] "Generic (PLEG): container finished" podID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerID="09c200898a8957afd5afb6fabd4f2d122082c3b5fe1b32efdfce8e0e10a7ddc5" exitCode=0 Jul 06 19:11:56.551810 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:56.551487 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" event={"ID":"08c50fa4-be66-43d0-8610-19a7e9111fcc","Type":"ContainerDied","Data":"09c200898a8957afd5afb6fabd4f2d122082c3b5fe1b32efdfce8e0e10a7ddc5"} Jul 06 19:11:56.553149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:56.553126 2569 generic.go:356] "Generic (PLEG): container finished" podID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerID="34f0adc7497f1bd5696e1a44aa8c7020da669f5cce21a7e0c41af0a02661f253" exitCode=0 Jul 06 19:11:56.553277 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:56.553158 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" event={"ID":"62b6ce61-ff82-4666-80fd-2f55a0ffecbc","Type":"ContainerDied","Data":"34f0adc7497f1bd5696e1a44aa8c7020da669f5cce21a7e0c41af0a02661f253"} Jul 06 19:11:56.554861 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:56.554841 2569 generic.go:356] "Generic (PLEG): container finished" podID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerID="7c4d6d32cccf97deabf30c93a4488feaab5dfd8edcd8a8982467738a06dbc0c5" exitCode=0 Jul 06 19:11:56.554958 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:56.554910 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" event={"ID":"933ceb0d-d79f-4e6f-bcd8-4659996a35ec","Type":"ContainerDied","Data":"7c4d6d32cccf97deabf30c93a4488feaab5dfd8edcd8a8982467738a06dbc0c5"} Jul 06 19:11:56.557177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:56.557074 2569 generic.go:356] "Generic (PLEG): container finished" podID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerID="e4564cae43ff9049922c5e1becc2e784b5ffeee783c70f7c22ab44f4706f9fbe" exitCode=0 Jul 06 19:11:56.557177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:56.557114 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" event={"ID":"8a6da782-ce13-47a7-81ad-e084cb1b9ccb","Type":"ContainerDied","Data":"e4564cae43ff9049922c5e1becc2e784b5ffeee783c70f7c22ab44f4706f9fbe"} Jul 06 19:11:57.564346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.564308 2569 generic.go:356] "Generic (PLEG): container finished" podID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerID="5ec7a74d5df90e52d5194784e218e61d27853f8ec6db58365524bb9dfc419e31" exitCode=0 Jul 06 19:11:57.564734 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.564355 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" event={"ID":"62b6ce61-ff82-4666-80fd-2f55a0ffecbc","Type":"ContainerDied","Data":"5ec7a74d5df90e52d5194784e218e61d27853f8ec6db58365524bb9dfc419e31"} Jul 06 19:11:57.566362 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.566341 2569 generic.go:356] "Generic (PLEG): container finished" podID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerID="acdcf31ed6a1e7b68d9c08ea6173f36f7350c799eb08d76620ea3ab1ada4e72a" exitCode=0 Jul 06 19:11:57.566488 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.566444 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" event={"ID":"933ceb0d-d79f-4e6f-bcd8-4659996a35ec","Type":"ContainerDied","Data":"acdcf31ed6a1e7b68d9c08ea6173f36f7350c799eb08d76620ea3ab1ada4e72a"} Jul 06 19:11:57.692620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.692592 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:57.722153 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.722131 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:57.783488 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.783460 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-util\") pod \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " Jul 06 19:11:57.783613 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.783566 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-bundle\") pod \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " Jul 06 19:11:57.783682 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.783635 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68b6q\" (UniqueName: \"kubernetes.io/projected/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-kube-api-access-68b6q\") pod \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\" (UID: \"8a6da782-ce13-47a7-81ad-e084cb1b9ccb\") " Jul 06 19:11:57.784184 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.784156 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-bundle" (OuterVolumeSpecName: "bundle") pod "8a6da782-ce13-47a7-81ad-e084cb1b9ccb" (UID: "8a6da782-ce13-47a7-81ad-e084cb1b9ccb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:57.785595 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.785567 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-kube-api-access-68b6q" (OuterVolumeSpecName: "kube-api-access-68b6q") pod "8a6da782-ce13-47a7-81ad-e084cb1b9ccb" (UID: "8a6da782-ce13-47a7-81ad-e084cb1b9ccb"). InnerVolumeSpecName "kube-api-access-68b6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:57.788790 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.788758 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-util" (OuterVolumeSpecName: "util") pod "8a6da782-ce13-47a7-81ad-e084cb1b9ccb" (UID: "8a6da782-ce13-47a7-81ad-e084cb1b9ccb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:57.884444 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.884370 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-bundle\") pod \"08c50fa4-be66-43d0-8610-19a7e9111fcc\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " Jul 06 19:11:57.884543 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.884469 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6l8r\" (UniqueName: \"kubernetes.io/projected/08c50fa4-be66-43d0-8610-19a7e9111fcc-kube-api-access-s6l8r\") pod \"08c50fa4-be66-43d0-8610-19a7e9111fcc\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " Jul 06 19:11:57.884543 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.884492 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-util\") pod \"08c50fa4-be66-43d0-8610-19a7e9111fcc\" (UID: \"08c50fa4-be66-43d0-8610-19a7e9111fcc\") " Jul 06 19:11:57.884646 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.884635 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-bundle\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:57.884684 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.884649 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68b6q\" (UniqueName: \"kubernetes.io/projected/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-kube-api-access-68b6q\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:57.884684 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.884657 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6da782-ce13-47a7-81ad-e084cb1b9ccb-util\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:57.884933 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.884912 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-bundle" (OuterVolumeSpecName: "bundle") pod "08c50fa4-be66-43d0-8610-19a7e9111fcc" (UID: "08c50fa4-be66-43d0-8610-19a7e9111fcc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:57.886454 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.886417 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c50fa4-be66-43d0-8610-19a7e9111fcc-kube-api-access-s6l8r" (OuterVolumeSpecName: "kube-api-access-s6l8r") pod "08c50fa4-be66-43d0-8610-19a7e9111fcc" (UID: "08c50fa4-be66-43d0-8610-19a7e9111fcc"). InnerVolumeSpecName "kube-api-access-s6l8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:57.889804 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.889786 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-util" (OuterVolumeSpecName: "util") pod "08c50fa4-be66-43d0-8610-19a7e9111fcc" (UID: "08c50fa4-be66-43d0-8610-19a7e9111fcc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:57.985571 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.985549 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6l8r\" (UniqueName: \"kubernetes.io/projected/08c50fa4-be66-43d0-8610-19a7e9111fcc-kube-api-access-s6l8r\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:57.985571 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.985567 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-util\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:57.985680 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:57.985576 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c50fa4-be66-43d0-8610-19a7e9111fcc-bundle\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:58.573875 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.573850 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" Jul 06 19:11:58.574262 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.573888 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q" event={"ID":"08c50fa4-be66-43d0-8610-19a7e9111fcc","Type":"ContainerDied","Data":"c8c4ef40a5a0a9781b058f7bdc51c97459bca75fd88088b1d2b1d1a77ce82985"} Jul 06 19:11:58.574262 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.573933 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8c4ef40a5a0a9781b058f7bdc51c97459bca75fd88088b1d2b1d1a77ce82985" Jul 06 19:11:58.575760 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.575743 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" Jul 06 19:11:58.575865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.575772 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z" event={"ID":"8a6da782-ce13-47a7-81ad-e084cb1b9ccb","Type":"ContainerDied","Data":"fcee4e85d1646caabbcc1df730babd0cd9cd81f29bb92d35e9439c07b4425078"} Jul 06 19:11:58.575865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.575806 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcee4e85d1646caabbcc1df730babd0cd9cd81f29bb92d35e9439c07b4425078" Jul 06 19:11:58.721185 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.721165 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:58.723140 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.723123 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:58.892318 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.892247 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-util\") pod \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " Jul 06 19:11:58.892318 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.892303 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlhls\" (UniqueName: \"kubernetes.io/projected/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-kube-api-access-nlhls\") pod \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " Jul 06 19:11:58.892516 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.892347 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gxdr\" (UniqueName: \"kubernetes.io/projected/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-kube-api-access-6gxdr\") pod \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " Jul 06 19:11:58.892516 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.892405 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-bundle\") pod \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\" (UID: \"933ceb0d-d79f-4e6f-bcd8-4659996a35ec\") " Jul 06 19:11:58.892516 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.892471 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-bundle\") pod \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " Jul 06 19:11:58.892668 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.892515 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-util\") pod \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\" (UID: \"62b6ce61-ff82-4666-80fd-2f55a0ffecbc\") " Jul 06 19:11:58.892955 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.892929 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-bundle" (OuterVolumeSpecName: "bundle") pod "62b6ce61-ff82-4666-80fd-2f55a0ffecbc" (UID: "62b6ce61-ff82-4666-80fd-2f55a0ffecbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:58.893074 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.892962 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-bundle" (OuterVolumeSpecName: "bundle") pod "933ceb0d-d79f-4e6f-bcd8-4659996a35ec" (UID: "933ceb0d-d79f-4e6f-bcd8-4659996a35ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:58.894588 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.894564 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-kube-api-access-nlhls" (OuterVolumeSpecName: "kube-api-access-nlhls") pod "933ceb0d-d79f-4e6f-bcd8-4659996a35ec" (UID: "933ceb0d-d79f-4e6f-bcd8-4659996a35ec"). InnerVolumeSpecName "kube-api-access-nlhls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:58.894714 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.894699 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-kube-api-access-6gxdr" (OuterVolumeSpecName: "kube-api-access-6gxdr") pod "62b6ce61-ff82-4666-80fd-2f55a0ffecbc" (UID: "62b6ce61-ff82-4666-80fd-2f55a0ffecbc"). InnerVolumeSpecName "kube-api-access-6gxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:58.898089 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.898067 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-util" (OuterVolumeSpecName: "util") pod "933ceb0d-d79f-4e6f-bcd8-4659996a35ec" (UID: "933ceb0d-d79f-4e6f-bcd8-4659996a35ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:58.898966 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.898879 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-util" (OuterVolumeSpecName: "util") pod "62b6ce61-ff82-4666-80fd-2f55a0ffecbc" (UID: "62b6ce61-ff82-4666-80fd-2f55a0ffecbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:58.993211 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.993189 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-util\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:58.993211 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.993208 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-util\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:58.993324 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.993218 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlhls\" (UniqueName: \"kubernetes.io/projected/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-kube-api-access-nlhls\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:58.993324 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.993228 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gxdr\" (UniqueName: \"kubernetes.io/projected/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-kube-api-access-6gxdr\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:58.993324 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.993237 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/933ceb0d-d79f-4e6f-bcd8-4659996a35ec-bundle\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:58.993324 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:58.993246 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62b6ce61-ff82-4666-80fd-2f55a0ffecbc-bundle\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:11:59.582082 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:59.582056 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" Jul 06 19:11:59.582082 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:59.582067 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22" event={"ID":"62b6ce61-ff82-4666-80fd-2f55a0ffecbc","Type":"ContainerDied","Data":"54110f07a087f9c5708a59595da5736aeb37e32d13a8387072930556da3c6bcb"} Jul 06 19:11:59.582540 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:59.582110 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54110f07a087f9c5708a59595da5736aeb37e32d13a8387072930556da3c6bcb" Jul 06 19:11:59.584073 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:59.584016 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" event={"ID":"933ceb0d-d79f-4e6f-bcd8-4659996a35ec","Type":"ContainerDied","Data":"a3add14d279f598b73185153a2d746033875b4b9035fb53aa434aa0f9ad5a427"} Jul 06 19:11:59.584184 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:59.584076 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m" Jul 06 19:11:59.584251 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:11:59.584215 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3add14d279f598b73185153a2d746033875b4b9035fb53aa434aa0f9ad5a427" Jul 06 19:12:00.900741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:00.900715 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:01.488090 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:01.488057 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:02.543855 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:02.543824 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:02.675628 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:02.675601 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:04.554842 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.554815 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx"] Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555128 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerName="util" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555138 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerName="util" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555146 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerName="pull" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555151 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerName="pull" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555160 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerName="util" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555166 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerName="util" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555174 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerName="extract" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555179 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerName="extract" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555184 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerName="pull" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555189 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerName="pull" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555198 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerName="util" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555203 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerName="util" Jul 06 19:12:04.555202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555207 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555213 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555223 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerName="util" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555227 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerName="util" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555234 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555239 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555246 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerName="pull" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555250 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerName="pull" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555255 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerName="pull" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555260 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerName="pull" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555265 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555269 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555321 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c50fa4-be66-43d0-8610-19a7e9111fcc" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555328 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="62b6ce61-ff82-4666-80fd-2f55a0ffecbc" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555334 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6da782-ce13-47a7-81ad-e084cb1b9ccb" containerName="extract" Jul 06 19:12:04.555578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.555340 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="933ceb0d-d79f-4e6f-bcd8-4659996a35ec" containerName="extract" Jul 06 19:12:04.558236 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.558220 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" Jul 06 19:12:04.561214 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.561193 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hpnkv\"" Jul 06 19:12:04.572174 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.572154 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx"] Jul 06 19:12:04.730982 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.730919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xpx\" (UniqueName: \"kubernetes.io/projected/7621b049-968c-4042-bcce-352cbf155c27-kube-api-access-59xpx\") pod \"authorino-operator-75cc8d5cb7-m2bvx\" (UID: \"7621b049-968c-4042-bcce-352cbf155c27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" Jul 06 19:12:04.831376 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.831320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59xpx\" (UniqueName: \"kubernetes.io/projected/7621b049-968c-4042-bcce-352cbf155c27-kube-api-access-59xpx\") pod \"authorino-operator-75cc8d5cb7-m2bvx\" (UID: \"7621b049-968c-4042-bcce-352cbf155c27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" Jul 06 19:12:04.842837 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.842813 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xpx\" (UniqueName: \"kubernetes.io/projected/7621b049-968c-4042-bcce-352cbf155c27-kube-api-access-59xpx\") pod \"authorino-operator-75cc8d5cb7-m2bvx\" (UID: \"7621b049-968c-4042-bcce-352cbf155c27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" Jul 06 19:12:04.868776 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.868754 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" Jul 06 19:12:04.992206 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:04.992183 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx"] Jul 06 19:12:04.994168 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:12:04.994145 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7621b049_968c_4042_bcce_352cbf155c27.slice/crio-d86794973fffea39d2467531df5b9265ad84343afb2eeab29ee59f2d8938073e WatchSource:0}: Error finding container d86794973fffea39d2467531df5b9265ad84343afb2eeab29ee59f2d8938073e: Status 404 returned error can't find the container with id d86794973fffea39d2467531df5b9265ad84343afb2eeab29ee59f2d8938073e Jul 06 19:12:05.618755 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:05.618723 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" event={"ID":"7621b049-968c-4042-bcce-352cbf155c27","Type":"ContainerStarted","Data":"d86794973fffea39d2467531df5b9265ad84343afb2eeab29ee59f2d8938073e"} Jul 06 19:12:05.919972 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:05.919864 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:08.185701 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.185622 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj"] Jul 06 19:12:08.189311 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.189289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:08.192370 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.192355 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rv5jx\"" Jul 06 19:12:08.204145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.204125 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj"] Jul 06 19:12:08.359135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.359102 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwm9\" (UniqueName: \"kubernetes.io/projected/1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a-kube-api-access-6nwm9\") pod \"limitador-operator-controller-manager-859d7b78bd-stdjj\" (UID: \"1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:08.459769 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.459701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwm9\" (UniqueName: \"kubernetes.io/projected/1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a-kube-api-access-6nwm9\") pod \"limitador-operator-controller-manager-859d7b78bd-stdjj\" (UID: \"1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:08.475270 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.475241 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwm9\" (UniqueName: \"kubernetes.io/projected/1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a-kube-api-access-6nwm9\") pod \"limitador-operator-controller-manager-859d7b78bd-stdjj\" (UID: \"1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:08.499226 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.499203 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:08.624241 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.624219 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj"] Jul 06 19:12:08.625747 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:12:08.625714 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa8fec4_f904_4ad0_bab4_06ef7fa7c52a.slice/crio-64d1d6b9c763bc2793d447061695d53441dad11af76ed2a550f971998cc6c1f2 WatchSource:0}: Error finding container 64d1d6b9c763bc2793d447061695d53441dad11af76ed2a550f971998cc6c1f2: Status 404 returned error can't find the container with id 64d1d6b9c763bc2793d447061695d53441dad11af76ed2a550f971998cc6c1f2 Jul 06 19:12:08.641252 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.641224 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" event={"ID":"7621b049-968c-4042-bcce-352cbf155c27","Type":"ContainerStarted","Data":"093d6b342632b9889f515237de3429cdf4eec93445f08aec254d060e6e06e089"} Jul 06 19:12:08.641373 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.641354 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" Jul 06 19:12:08.642410 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.642393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" event={"ID":"1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a","Type":"ContainerStarted","Data":"64d1d6b9c763bc2793d447061695d53441dad11af76ed2a550f971998cc6c1f2"} Jul 06 19:12:08.660737 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.660697 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" podStartSLOduration=1.836119085 podStartE2EDuration="4.660684888s" totalImagesPullingTime="2.824565803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:08.659803476 +0000 UTC m=+356.200525704" watchObservedRunningTime="2026-07-06 19:12:08.660684888 +0000 UTC m=+356.201407182" Jul 06 19:12:08.896962 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:08.896938 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:10.403384 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:10.403357 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:11.660329 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:11.660294 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" event={"ID":"1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a","Type":"ContainerStarted","Data":"9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee"} Jul 06 19:12:11.660698 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:11.660394 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:11.683663 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:11.683617 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" podStartSLOduration=1.169174249 podStartE2EDuration="3.683601514s" totalImagesPullingTime="2.514427265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:11.681745512 +0000 UTC m=+359.222467739" watchObservedRunningTime="2026-07-06 19:12:11.683601514 +0000 UTC m=+359.224323835" Jul 06 19:12:11.777670 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:11.777640 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:12.166992 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.166961 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8"] Jul 06 19:12:12.170271 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.170256 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:12.172817 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.172797 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vpnh8\"" Jul 06 19:12:12.185437 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.185408 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8"] Jul 06 19:12:12.295107 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.295084 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" (UID: \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:12.295243 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.295121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jzg\" (UniqueName: \"kubernetes.io/projected/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-kube-api-access-s5jzg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" (UID: \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:12.395518 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.395495 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" (UID: \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:12.395659 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.395531 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jzg\" (UniqueName: \"kubernetes.io/projected/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-kube-api-access-s5jzg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" (UID: \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:12.395905 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.395885 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" (UID: \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:12.410979 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.410948 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jzg\" (UniqueName: \"kubernetes.io/projected/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-kube-api-access-s5jzg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" (UID: \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:12.480730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.480665 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:12.616822 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.616795 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8"] Jul 06 19:12:12.617764 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:12:12.617739 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4aabaf3_50f0_4bba_94e0_4d55d505d81f.slice/crio-eed1bb09a208246ba54729a6d6138f98cbfb90739093fbbb944a127061674a17 WatchSource:0}: Error finding container eed1bb09a208246ba54729a6d6138f98cbfb90739093fbbb944a127061674a17: Status 404 returned error can't find the container with id eed1bb09a208246ba54729a6d6138f98cbfb90739093fbbb944a127061674a17 Jul 06 19:12:12.665671 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:12.665642 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" event={"ID":"a4aabaf3-50f0-4bba-94e0-4d55d505d81f","Type":"ContainerStarted","Data":"eed1bb09a208246ba54729a6d6138f98cbfb90739093fbbb944a127061674a17"} Jul 06 19:12:13.128776 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:13.128703 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:14.529801 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:14.529772 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:16.328975 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:16.328941 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:16.427124 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:16.427100 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:16.726028 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:16.725951 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:16.833236 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:16.833205 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:17.426197 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:17.426162 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:17.536371 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:17.536333 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:19.650876 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:19.650859 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-m2bvx" Jul 06 19:12:20.702443 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:20.702390 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:20.717193 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:20.717164 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" event={"ID":"a4aabaf3-50f0-4bba-94e0-4d55d505d81f","Type":"ContainerStarted","Data":"8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5"} Jul 06 19:12:20.717324 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:20.717261 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:20.742568 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:20.742522 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" podStartSLOduration=1.718393273 podStartE2EDuration="8.742508956s" totalImagesPullingTime="7.024115683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:20.739629815 +0000 UTC m=+368.280352065" watchObservedRunningTime="2026-07-06 19:12:20.742508956 +0000 UTC m=+368.283231183" Jul 06 19:12:20.808107 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:20.808078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:22.267533 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.267498 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn"] Jul 06 19:12:22.271327 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.271307 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.273167 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.273142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.273282 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.273236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.273282 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.273269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8vh2\" (UniqueName: \"kubernetes.io/projected/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-kube-api-access-n8vh2\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.273829 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.273813 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 19:12:22.273926 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.273832 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n82z9\"" Jul 06 19:12:22.273926 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.273835 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 19:12:22.281681 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.281658 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn"] Jul 06 19:12:22.350567 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.350542 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:22.374081 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.374058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.374185 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.374095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8vh2\" (UniqueName: \"kubernetes.io/projected/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-kube-api-access-n8vh2\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.374185 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.374145 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.374292 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:22.374260 2569 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 19:12:22.374348 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:22.374320 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-plugin-serving-cert podName:b5b9dafb-e8e0-46f4-b451-e03dd281dc84 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:22.874302198 +0000 UTC m=+370.415024405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-jtrtn" (UID: "b5b9dafb-e8e0-46f4-b451-e03dd281dc84") : secret "plugin-serving-cert" not found Jul 06 19:12:22.374705 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.374687 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.384485 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.384467 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8vh2\" (UniqueName: \"kubernetes.io/projected/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-kube-api-access-n8vh2\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.668181 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.668129 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:22.721618 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.721595 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:22.876987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.876961 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.879146 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.879129 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9dafb-e8e0-46f4-b451-e03dd281dc84-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jtrtn\" (UID: \"b5b9dafb-e8e0-46f4-b451-e03dd281dc84\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:22.879960 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:22.879947 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" Jul 06 19:12:23.018536 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:12:23.018507 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5b9dafb_e8e0_46f4_b451_e03dd281dc84.slice/crio-205bea0fea62a0d7b65ce2d06f81801eee1262a695221404a2a53ebb36aed685 WatchSource:0}: Error finding container 205bea0fea62a0d7b65ce2d06f81801eee1262a695221404a2a53ebb36aed685: Status 404 returned error can't find the container with id 205bea0fea62a0d7b65ce2d06f81801eee1262a695221404a2a53ebb36aed685 Jul 06 19:12:23.022396 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:23.022373 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn"] Jul 06 19:12:23.733081 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:23.733049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" event={"ID":"b5b9dafb-e8e0-46f4-b451-e03dd281dc84","Type":"ContainerStarted","Data":"205bea0fea62a0d7b65ce2d06f81801eee1262a695221404a2a53ebb36aed685"} Jul 06 19:12:24.153884 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:24.153854 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:24.253868 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:24.253832 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:27.957538 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:27.957493 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:28.057960 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:28.057931 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:28.159175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:28.159135 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:30.617165 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:30.617130 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:30.717392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:30.717366 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:31.724724 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:31.724696 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.403771 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.403740 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8"] Jul 06 19:12:33.404450 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.404391 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" containerName="manager" containerID="cri-o://8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5" gracePeriod=2 Jul 06 19:12:33.409589 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.409553 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.413597 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.412026 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8"] Jul 06 19:12:33.427595 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.427567 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj"] Jul 06 19:12:33.427886 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.427815 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" containerName="manager" containerID="cri-o://9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee" gracePeriod=2 Jul 06 19:12:33.435813 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.435755 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj"] Jul 06 19:12:33.437789 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.437767 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f"] Jul 06 19:12:33.438136 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.438120 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" containerName="manager" Jul 06 19:12:33.438177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.438140 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" containerName="manager" Jul 06 19:12:33.438177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.438160 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" containerName="manager" Jul 06 19:12:33.438177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.438169 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" containerName="manager" Jul 06 19:12:33.438266 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.438236 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" containerName="manager" Jul 06 19:12:33.438266 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.438245 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" containerName="manager" Jul 06 19:12:33.440266 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.440242 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.441334 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.441320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:33.443933 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.443910 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stdjj\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.446020 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.445994 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.450476 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.450455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cp7t\" (UniqueName: \"kubernetes.io/projected/73c26a17-1760-4988-ba8f-2e0d125a0185-kube-api-access-9cp7t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wv79f\" (UID: \"73c26a17-1760-4988-ba8f-2e0d125a0185\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:33.450552 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.450527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73c26a17-1760-4988-ba8f-2e0d125a0185-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wv79f\" (UID: \"73c26a17-1760-4988-ba8f-2e0d125a0185\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:33.460538 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.460515 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j"] Jul 06 19:12:33.465335 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.465265 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f"] Jul 06 19:12:33.465463 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.465366 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" Jul 06 19:12:33.490343 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.490311 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stdjj\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.491038 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.490983 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j"] Jul 06 19:12:33.493093 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.493032 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.552858 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.552828 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73c26a17-1760-4988-ba8f-2e0d125a0185-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wv79f\" (UID: \"73c26a17-1760-4988-ba8f-2e0d125a0185\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:33.552980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.552912 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cp7t\" (UniqueName: \"kubernetes.io/projected/73c26a17-1760-4988-ba8f-2e0d125a0185-kube-api-access-9cp7t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wv79f\" (UID: \"73c26a17-1760-4988-ba8f-2e0d125a0185\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:33.552980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.552946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg44r\" (UniqueName: \"kubernetes.io/projected/82fc05a1-f092-4468-bd99-b9b9a2064399-kube-api-access-qg44r\") pod \"limitador-operator-controller-manager-859d7b78bd-ddp6j\" (UID: \"82fc05a1-f092-4468-bd99-b9b9a2064399\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" Jul 06 19:12:33.553237 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.553215 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73c26a17-1760-4988-ba8f-2e0d125a0185-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wv79f\" (UID: \"73c26a17-1760-4988-ba8f-2e0d125a0185\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:33.571768 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.571740 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cp7t\" (UniqueName: \"kubernetes.io/projected/73c26a17-1760-4988-ba8f-2e0d125a0185-kube-api-access-9cp7t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wv79f\" (UID: \"73c26a17-1760-4988-ba8f-2e0d125a0185\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:33.654442 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.654331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg44r\" (UniqueName: \"kubernetes.io/projected/82fc05a1-f092-4468-bd99-b9b9a2064399-kube-api-access-qg44r\") pod \"limitador-operator-controller-manager-859d7b78bd-ddp6j\" (UID: \"82fc05a1-f092-4468-bd99-b9b9a2064399\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" Jul 06 19:12:33.666991 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.666972 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg44r\" (UniqueName: \"kubernetes.io/projected/82fc05a1-f092-4468-bd99-b9b9a2064399-kube-api-access-qg44r\") pod \"limitador-operator-controller-manager-859d7b78bd-ddp6j\" (UID: \"82fc05a1-f092-4468-bd99-b9b9a2064399\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" Jul 06 19:12:33.673966 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.673941 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.675939 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.675924 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.676730 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.676708 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stdjj\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.679407 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.679390 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.683913 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.683894 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stdjj\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.685992 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.685965 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.755665 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.755639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nwm9\" (UniqueName: \"kubernetes.io/projected/1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a-kube-api-access-6nwm9\") pod \"1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a\" (UID: \"1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a\") " Jul 06 19:12:33.755833 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.755713 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-extensions-socket-volume\") pod \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\" (UID: \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\") " Jul 06 19:12:33.755833 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.755745 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5jzg\" (UniqueName: \"kubernetes.io/projected/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-kube-api-access-s5jzg\") pod \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\" (UID: \"a4aabaf3-50f0-4bba-94e0-4d55d505d81f\") " Jul 06 19:12:33.756233 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.756206 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a4aabaf3-50f0-4bba-94e0-4d55d505d81f" (UID: "a4aabaf3-50f0-4bba-94e0-4d55d505d81f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:12:33.757875 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.757837 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-kube-api-access-s5jzg" (OuterVolumeSpecName: "kube-api-access-s5jzg") pod "a4aabaf3-50f0-4bba-94e0-4d55d505d81f" (UID: "a4aabaf3-50f0-4bba-94e0-4d55d505d81f"). InnerVolumeSpecName "kube-api-access-s5jzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:12:33.757995 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.757920 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a-kube-api-access-6nwm9" (OuterVolumeSpecName: "kube-api-access-6nwm9") pod "1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" (UID: "1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a"). InnerVolumeSpecName "kube-api-access-6nwm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:12:33.797710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.797687 2569 generic.go:356] "Generic (PLEG): container finished" podID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" containerID="9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee" exitCode=0 Jul 06 19:12:33.797817 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.797727 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.797817 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.797779 2569 scope.go:117] "RemoveContainer" containerID="9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee" Jul 06 19:12:33.799681 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.799650 2569 generic.go:356] "Generic (PLEG): container finished" podID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" containerID="8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5" exitCode=0 Jul 06 19:12:33.799793 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.799775 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.802493 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.802469 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.804969 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.804938 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stdjj\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.807680 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.807643 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stdjj\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.810073 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.810048 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.813623 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.813597 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stdjj\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.815907 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.815885 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.818278 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.818256 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-stdjj\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-stdjj" Jul 06 19:12:33.820722 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:33.820694 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-58km8\" is forbidden: User \"system:node:ip-10-0-131-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-1.ec2.internal' and this object" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-58km8" Jul 06 19:12:33.829471 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.829448 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:33.836823 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.836803 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" Jul 06 19:12:33.856438 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.856396 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-extensions-socket-volume\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:12:33.856537 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.856450 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jzg\" (UniqueName: \"kubernetes.io/projected/a4aabaf3-50f0-4bba-94e0-4d55d505d81f-kube-api-access-s5jzg\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:12:33.856537 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:33.856466 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nwm9\" (UniqueName: \"kubernetes.io/projected/1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a-kube-api-access-6nwm9\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:12:35.023521 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:35.023490 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a" path="/var/lib/kubelet/pods/1aa8fec4-f904-4ad0-bab4-06ef7fa7c52a/volumes" Jul 06 19:12:35.023945 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:35.023930 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4aabaf3-50f0-4bba-94e0-4d55d505d81f" path="/var/lib/kubelet/pods/a4aabaf3-50f0-4bba-94e0-4d55d505d81f/volumes" Jul 06 19:12:36.390607 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:36.390574 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:36.493245 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:36.493214 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:36.999285 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:36.999239 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:37.099203 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:37.099167 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:40.407449 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:40.407392 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:40.509413 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:40.509336 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:43.074967 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:43.074927 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:43.222965 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:43.222899 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:46.863142 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:46.863004 2569 scope.go:117] "RemoveContainer" containerID="9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee" Jul 06 19:12:46.863568 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:46.863539 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee\": container with ID starting with 9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee not found: ID does not exist" containerID="9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee" Jul 06 19:12:46.863640 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:46.863585 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee"} err="failed to get container status \"9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee\": rpc error: code = NotFound desc = could not find container \"9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee\": container with ID starting with 9397c3511c667bc35ae487564d36afd99db5cd6e2f3b50ada6b84953173d1cee not found: ID does not exist" Jul 06 19:12:46.863640 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:46.863615 2569 scope.go:117] "RemoveContainer" containerID="8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5" Jul 06 19:12:46.899568 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:46.899525 2569 scope.go:117] "RemoveContainer" containerID="8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5" Jul 06 19:12:46.899835 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:46.899818 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5\": container with ID starting with 8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5 not found: ID does not exist" containerID="8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5" Jul 06 19:12:46.899957 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:46.899846 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5"} err="failed to get container status \"8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5\": rpc error: code = NotFound desc = could not find container \"8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5\": container with ID starting with 8966279f739b16d6e5b158caf29848fb583681f1f8fc1eb63617eceaf5c834f5 not found: ID does not exist" Jul 06 19:12:47.037375 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.037351 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f"] Jul 06 19:12:47.038943 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:12:47.038922 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c26a17_1760_4988_ba8f_2e0d125a0185.slice/crio-2db2978e6a963c230b2c691ad141bf81f25d539eab4ae7c5e358355de5fca757 WatchSource:0}: Error finding container 2db2978e6a963c230b2c691ad141bf81f25d539eab4ae7c5e358355de5fca757: Status 404 returned error can't find the container with id 2db2978e6a963c230b2c691ad141bf81f25d539eab4ae7c5e358355de5fca757 Jul 06 19:12:47.061940 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.061917 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j"] Jul 06 19:12:47.065437 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:12:47.065403 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fc05a1_f092_4468_bd99_b9b9a2064399.slice/crio-f7f30386b87e41fb2c44b5098add60b1530da67089eca84c79455d09c3d42bfb WatchSource:0}: Error finding container f7f30386b87e41fb2c44b5098add60b1530da67089eca84c79455d09c3d42bfb: Status 404 returned error can't find the container with id f7f30386b87e41fb2c44b5098add60b1530da67089eca84c79455d09c3d42bfb Jul 06 19:12:47.887200 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.887157 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" event={"ID":"73c26a17-1760-4988-ba8f-2e0d125a0185","Type":"ContainerStarted","Data":"7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d"} Jul 06 19:12:47.887637 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.887221 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" event={"ID":"73c26a17-1760-4988-ba8f-2e0d125a0185","Type":"ContainerStarted","Data":"2db2978e6a963c230b2c691ad141bf81f25d539eab4ae7c5e358355de5fca757"} Jul 06 19:12:47.887637 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.887314 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:47.888655 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.888628 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" event={"ID":"b5b9dafb-e8e0-46f4-b451-e03dd281dc84","Type":"ContainerStarted","Data":"ae972ebee4ced00b61353cda40d1dfef13f36672a364837d8087ba5ad6ce2111"} Jul 06 19:12:47.890913 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.890894 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" event={"ID":"82fc05a1-f092-4468-bd99-b9b9a2064399","Type":"ContainerStarted","Data":"efa2d1278b33044e4d2e32c0ea4f7b12cb21974087219bcfe65e84aeba3cff4a"} Jul 06 19:12:47.891031 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.890927 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" event={"ID":"82fc05a1-f092-4468-bd99-b9b9a2064399","Type":"ContainerStarted","Data":"f7f30386b87e41fb2c44b5098add60b1530da67089eca84c79455d09c3d42bfb"} Jul 06 19:12:47.891031 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.890949 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" Jul 06 19:12:47.908197 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.908154 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" podStartSLOduration=14.908141942 podStartE2EDuration="14.908141942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:47.905818737 +0000 UTC m=+395.446540965" watchObservedRunningTime="2026-07-06 19:12:47.908141942 +0000 UTC m=+395.448864169" Jul 06 19:12:47.927718 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.927682 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtrtn" podStartSLOduration=2.002496002 podStartE2EDuration="25.927669298s" totalImagesPullingTime="23.925173296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:47.926726919 +0000 UTC m=+395.467449146" watchObservedRunningTime="2026-07-06 19:12:47.927669298 +0000 UTC m=+395.468391526" Jul 06 19:12:47.944987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:47.944953 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" podStartSLOduration=14.944941171 podStartE2EDuration="14.944941171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:47.943441189 +0000 UTC m=+395.484163408" watchObservedRunningTime="2026-07-06 19:12:47.944941171 +0000 UTC m=+395.485663399" Jul 06 19:12:49.511614 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:49.511577 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:49.608670 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:49.608642 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:50.926251 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:50.926213 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f"] Jul 06 19:12:50.926789 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:50.926446 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" podUID="73c26a17-1760-4988-ba8f-2e0d125a0185" containerName="manager" containerID="cri-o://7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d" gracePeriod=10 Jul 06 19:12:51.165959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.165937 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:51.173196 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.173173 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz"] Jul 06 19:12:51.173706 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.173692 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c26a17-1760-4988-ba8f-2e0d125a0185" containerName="manager" Jul 06 19:12:51.173755 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.173710 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c26a17-1760-4988-ba8f-2e0d125a0185" containerName="manager" Jul 06 19:12:51.173834 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.173823 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c26a17-1760-4988-ba8f-2e0d125a0185" containerName="manager" Jul 06 19:12:51.175854 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.175839 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.199332 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.199310 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cp7t\" (UniqueName: \"kubernetes.io/projected/73c26a17-1760-4988-ba8f-2e0d125a0185-kube-api-access-9cp7t\") pod \"73c26a17-1760-4988-ba8f-2e0d125a0185\" (UID: \"73c26a17-1760-4988-ba8f-2e0d125a0185\") " Jul 06 19:12:51.199469 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.199349 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73c26a17-1760-4988-ba8f-2e0d125a0185-extensions-socket-volume\") pod \"73c26a17-1760-4988-ba8f-2e0d125a0185\" (UID: \"73c26a17-1760-4988-ba8f-2e0d125a0185\") " Jul 06 19:12:51.199763 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.199735 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c26a17-1760-4988-ba8f-2e0d125a0185-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "73c26a17-1760-4988-ba8f-2e0d125a0185" (UID: "73c26a17-1760-4988-ba8f-2e0d125a0185"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:12:51.201156 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.201131 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c26a17-1760-4988-ba8f-2e0d125a0185-kube-api-access-9cp7t" (OuterVolumeSpecName: "kube-api-access-9cp7t") pod "73c26a17-1760-4988-ba8f-2e0d125a0185" (UID: "73c26a17-1760-4988-ba8f-2e0d125a0185"). InnerVolumeSpecName "kube-api-access-9cp7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:12:51.208572 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.208552 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz"] Jul 06 19:12:51.300285 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.300259 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllgq\" (UniqueName: \"kubernetes.io/projected/67b91cd9-892a-4e61-9a3c-a23e09926a07-kube-api-access-vllgq\") pod \"kuadrant-operator-controller-manager-86878c769f-pgrlz\" (UID: \"67b91cd9-892a-4e61-9a3c-a23e09926a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.300378 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.300320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b91cd9-892a-4e61-9a3c-a23e09926a07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pgrlz\" (UID: \"67b91cd9-892a-4e61-9a3c-a23e09926a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.300456 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.300386 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cp7t\" (UniqueName: \"kubernetes.io/projected/73c26a17-1760-4988-ba8f-2e0d125a0185-kube-api-access-9cp7t\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:12:51.300456 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.300404 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73c26a17-1760-4988-ba8f-2e0d125a0185-extensions-socket-volume\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:12:51.400940 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.400916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllgq\" (UniqueName: \"kubernetes.io/projected/67b91cd9-892a-4e61-9a3c-a23e09926a07-kube-api-access-vllgq\") pod \"kuadrant-operator-controller-manager-86878c769f-pgrlz\" (UID: \"67b91cd9-892a-4e61-9a3c-a23e09926a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.401032 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.400993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b91cd9-892a-4e61-9a3c-a23e09926a07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pgrlz\" (UID: \"67b91cd9-892a-4e61-9a3c-a23e09926a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.401338 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.401322 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b91cd9-892a-4e61-9a3c-a23e09926a07-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pgrlz\" (UID: \"67b91cd9-892a-4e61-9a3c-a23e09926a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.412976 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.412957 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllgq\" (UniqueName: \"kubernetes.io/projected/67b91cd9-892a-4e61-9a3c-a23e09926a07-kube-api-access-vllgq\") pod \"kuadrant-operator-controller-manager-86878c769f-pgrlz\" (UID: \"67b91cd9-892a-4e61-9a3c-a23e09926a07\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.484553 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.484500 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.611053 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.611029 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz"] Jul 06 19:12:51.612560 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:12:51.612536 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b91cd9_892a_4e61_9a3c_a23e09926a07.slice/crio-a420f391149a97aebe07191cc987141a323ca5197f8e6f8d516e8dca7e5b3b78 WatchSource:0}: Error finding container a420f391149a97aebe07191cc987141a323ca5197f8e6f8d516e8dca7e5b3b78: Status 404 returned error can't find the container with id a420f391149a97aebe07191cc987141a323ca5197f8e6f8d516e8dca7e5b3b78 Jul 06 19:12:51.911912 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.911881 2569 generic.go:356] "Generic (PLEG): container finished" podID="73c26a17-1760-4988-ba8f-2e0d125a0185" containerID="7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d" exitCode=0 Jul 06 19:12:51.912086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.911925 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" Jul 06 19:12:51.912086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.911965 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" event={"ID":"73c26a17-1760-4988-ba8f-2e0d125a0185","Type":"ContainerDied","Data":"7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d"} Jul 06 19:12:51.912086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.912025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f" event={"ID":"73c26a17-1760-4988-ba8f-2e0d125a0185","Type":"ContainerDied","Data":"2db2978e6a963c230b2c691ad141bf81f25d539eab4ae7c5e358355de5fca757"} Jul 06 19:12:51.912086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.912057 2569 scope.go:117] "RemoveContainer" containerID="7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d" Jul 06 19:12:51.913745 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.913720 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" event={"ID":"67b91cd9-892a-4e61-9a3c-a23e09926a07","Type":"ContainerStarted","Data":"3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17"} Jul 06 19:12:51.913860 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.913765 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" event={"ID":"67b91cd9-892a-4e61-9a3c-a23e09926a07","Type":"ContainerStarted","Data":"a420f391149a97aebe07191cc987141a323ca5197f8e6f8d516e8dca7e5b3b78"} Jul 06 19:12:51.913860 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.913800 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:12:51.921195 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.921178 2569 scope.go:117] "RemoveContainer" containerID="7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d" Jul 06 19:12:51.921467 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:12:51.921445 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d\": container with ID starting with 7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d not found: ID does not exist" containerID="7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d" Jul 06 19:12:51.921565 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.921474 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d"} err="failed to get container status \"7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d\": rpc error: code = NotFound desc = could not find container \"7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d\": container with ID starting with 7b89d985aacded01f830f510c5d2d7f34570919f4aa0b0fa32f1f4586436461d not found: ID does not exist" Jul 06 19:12:51.943079 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.942805 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" podStartSLOduration=0.942794134 podStartE2EDuration="942.794134ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:51.942634146 +0000 UTC m=+399.483356376" watchObservedRunningTime="2026-07-06 19:12:51.942794134 +0000 UTC m=+399.483516361" Jul 06 19:12:51.962208 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.962189 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f"] Jul 06 19:12:51.965886 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:51.965869 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wv79f"] Jul 06 19:12:53.021741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:53.021713 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c26a17-1760-4988-ba8f-2e0d125a0185" path="/var/lib/kubelet/pods/73c26a17-1760-4988-ba8f-2e0d125a0185/volumes" Jul 06 19:12:58.897678 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:12:58.897653 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ddp6j" Jul 06 19:13:02.923520 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:02.923489 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:13:06.118833 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:06.118794 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:07.352201 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:07.352173 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:07.606101 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:07.606026 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:19.747212 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:19.747181 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:19.852836 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:19.852798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:19.921648 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:19.921618 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:20.235646 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.235615 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:13:20.268636 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.268605 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:13:20.268792 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.268699 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:20.270863 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.270844 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:13:20.274949 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.274930 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:13:20.423022 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.422995 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/543b67bc-9f9c-416f-86c5-4c1b391bae04-config-file\") pod \"limitador-limitador-54d6d988d5-plldc\" (UID: \"543b67bc-9f9c-416f-86c5-4c1b391bae04\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:20.423147 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.423044 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnnjb\" (UniqueName: \"kubernetes.io/projected/543b67bc-9f9c-416f-86c5-4c1b391bae04-kube-api-access-nnnjb\") pod \"limitador-limitador-54d6d988d5-plldc\" (UID: \"543b67bc-9f9c-416f-86c5-4c1b391bae04\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:20.524310 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.524280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnnjb\" (UniqueName: \"kubernetes.io/projected/543b67bc-9f9c-416f-86c5-4c1b391bae04-kube-api-access-nnnjb\") pod \"limitador-limitador-54d6d988d5-plldc\" (UID: \"543b67bc-9f9c-416f-86c5-4c1b391bae04\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:20.524491 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.524380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/543b67bc-9f9c-416f-86c5-4c1b391bae04-config-file\") pod \"limitador-limitador-54d6d988d5-plldc\" (UID: \"543b67bc-9f9c-416f-86c5-4c1b391bae04\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:20.525119 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.525098 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/543b67bc-9f9c-416f-86c5-4c1b391bae04-config-file\") pod \"limitador-limitador-54d6d988d5-plldc\" (UID: \"543b67bc-9f9c-416f-86c5-4c1b391bae04\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:20.538838 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.538813 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnnjb\" (UniqueName: \"kubernetes.io/projected/543b67bc-9f9c-416f-86c5-4c1b391bae04-kube-api-access-nnnjb\") pod \"limitador-limitador-54d6d988d5-plldc\" (UID: \"543b67bc-9f9c-416f-86c5-4c1b391bae04\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:20.543700 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.543678 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-blzqc"] Jul 06 19:13:20.547032 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.547016 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-blzqc" Jul 06 19:13:20.548949 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.548935 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dbgbq\"" Jul 06 19:13:20.554779 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.554758 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-blzqc"] Jul 06 19:13:20.579021 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.578999 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:20.593772 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.593752 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:20.625645 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.625081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgzzj\" (UniqueName: \"kubernetes.io/projected/071c63bb-2594-44b3-853d-20bd8ff57c77-kube-api-access-vgzzj\") pod \"authorino-58cd955675-blzqc\" (UID: \"071c63bb-2594-44b3-853d-20bd8ff57c77\") " pod="kuadrant-system/authorino-58cd955675-blzqc" Jul 06 19:13:20.713729 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.713703 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:20.724538 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.724507 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:13:20.724935 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:13:20.724912 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543b67bc_9f9c_416f_86c5_4c1b391bae04.slice/crio-c78f725d5fdfb109f8040b6360917b324170a8f93cb2d6634d8c5cb0f4b6f259 WatchSource:0}: Error finding container c78f725d5fdfb109f8040b6360917b324170a8f93cb2d6634d8c5cb0f4b6f259: Status 404 returned error can't find the container with id c78f725d5fdfb109f8040b6360917b324170a8f93cb2d6634d8c5cb0f4b6f259 Jul 06 19:13:20.725924 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.725904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgzzj\" (UniqueName: \"kubernetes.io/projected/071c63bb-2594-44b3-853d-20bd8ff57c77-kube-api-access-vgzzj\") pod \"authorino-58cd955675-blzqc\" (UID: \"071c63bb-2594-44b3-853d-20bd8ff57c77\") " pod="kuadrant-system/authorino-58cd955675-blzqc" Jul 06 19:13:20.742877 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.742857 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgzzj\" (UniqueName: \"kubernetes.io/projected/071c63bb-2594-44b3-853d-20bd8ff57c77-kube-api-access-vgzzj\") pod \"authorino-58cd955675-blzqc\" (UID: \"071c63bb-2594-44b3-853d-20bd8ff57c77\") " pod="kuadrant-system/authorino-58cd955675-blzqc" Jul 06 19:13:20.785581 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.785532 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-tgkxt"] Jul 06 19:13:20.789572 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.789556 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tgkxt" Jul 06 19:13:20.795075 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.795043 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tgkxt"] Jul 06 19:13:20.856656 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.856636 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-blzqc" Jul 06 19:13:20.927918 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.927893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b862t\" (UniqueName: \"kubernetes.io/projected/3b9efed4-c5d3-4309-91c8-8e8bd992f08d-kube-api-access-b862t\") pod \"authorino-75d556bd77-tgkxt\" (UID: \"3b9efed4-c5d3-4309-91c8-8e8bd992f08d\") " pod="kuadrant-system/authorino-75d556bd77-tgkxt" Jul 06 19:13:20.979032 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:20.979008 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-blzqc"] Jul 06 19:13:20.980537 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:13:20.980516 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod071c63bb_2594_44b3_853d_20bd8ff57c77.slice/crio-a7f8b51059d7933be23de3ee11dc5c8c26150249d718e3761b4e833077ce0f8a WatchSource:0}: Error finding container a7f8b51059d7933be23de3ee11dc5c8c26150249d718e3761b4e833077ce0f8a: Status 404 returned error can't find the container with id a7f8b51059d7933be23de3ee11dc5c8c26150249d718e3761b4e833077ce0f8a Jul 06 19:13:21.028554 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:21.028534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b862t\" (UniqueName: \"kubernetes.io/projected/3b9efed4-c5d3-4309-91c8-8e8bd992f08d-kube-api-access-b862t\") pod \"authorino-75d556bd77-tgkxt\" (UID: \"3b9efed4-c5d3-4309-91c8-8e8bd992f08d\") " pod="kuadrant-system/authorino-75d556bd77-tgkxt" Jul 06 19:13:21.036364 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:21.036323 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b862t\" (UniqueName: \"kubernetes.io/projected/3b9efed4-c5d3-4309-91c8-8e8bd992f08d-kube-api-access-b862t\") pod \"authorino-75d556bd77-tgkxt\" (UID: \"3b9efed4-c5d3-4309-91c8-8e8bd992f08d\") " pod="kuadrant-system/authorino-75d556bd77-tgkxt" Jul 06 19:13:21.074277 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:21.074255 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" event={"ID":"543b67bc-9f9c-416f-86c5-4c1b391bae04","Type":"ContainerStarted","Data":"c78f725d5fdfb109f8040b6360917b324170a8f93cb2d6634d8c5cb0f4b6f259"} Jul 06 19:13:21.075339 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:21.075318 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-blzqc" event={"ID":"071c63bb-2594-44b3-853d-20bd8ff57c77","Type":"ContainerStarted","Data":"a7f8b51059d7933be23de3ee11dc5c8c26150249d718e3761b4e833077ce0f8a"} Jul 06 19:13:21.099510 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:21.099492 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tgkxt" Jul 06 19:13:21.246727 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:21.246696 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tgkxt"] Jul 06 19:13:21.248101 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:13:21.248071 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b9efed4_c5d3_4309_91c8_8e8bd992f08d.slice/crio-ed2192948a3ffaf68acbdf88d97b7f399530e7282732a70159fb2790416c147a WatchSource:0}: Error finding container ed2192948a3ffaf68acbdf88d97b7f399530e7282732a70159fb2790416c147a: Status 404 returned error can't find the container with id ed2192948a3ffaf68acbdf88d97b7f399530e7282732a70159fb2790416c147a Jul 06 19:13:22.095855 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:22.095819 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tgkxt" event={"ID":"3b9efed4-c5d3-4309-91c8-8e8bd992f08d","Type":"ContainerStarted","Data":"ed2192948a3ffaf68acbdf88d97b7f399530e7282732a70159fb2790416c147a"} Jul 06 19:13:26.124356 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:26.124318 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-blzqc" event={"ID":"071c63bb-2594-44b3-853d-20bd8ff57c77","Type":"ContainerStarted","Data":"5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85"} Jul 06 19:13:26.125936 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:26.125911 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tgkxt" event={"ID":"3b9efed4-c5d3-4309-91c8-8e8bd992f08d","Type":"ContainerStarted","Data":"56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995"} Jul 06 19:13:26.127644 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:26.127619 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" event={"ID":"543b67bc-9f9c-416f-86c5-4c1b391bae04","Type":"ContainerStarted","Data":"4b17846b69827a43e3c0822623041af894d414327c993093185d36759b8b2d48"} Jul 06 19:13:26.127746 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:26.127735 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:26.139734 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:26.139691 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-blzqc" podStartSLOduration=1.48824619 podStartE2EDuration="6.139678431s" totalImagesPullingTime="4.651432241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:26.138274962 +0000 UTC m=+433.678997190" watchObservedRunningTime="2026-07-06 19:13:26.139678431 +0000 UTC m=+433.680400658" Jul 06 19:13:26.154171 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:26.154128 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" podStartSLOduration=1.238661004 podStartE2EDuration="6.154111983s" totalImagesPullingTime="4.915450979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:26.153111534 +0000 UTC m=+433.693833774" watchObservedRunningTime="2026-07-06 19:13:26.154111983 +0000 UTC m=+433.694834210" Jul 06 19:13:26.167815 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:26.167762 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-tgkxt" podStartSLOduration=1.814705559 podStartE2EDuration="6.16774515s" totalImagesPullingTime="4.353039591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:26.165778846 +0000 UTC m=+433.706501075" watchObservedRunningTime="2026-07-06 19:13:26.16774515 +0000 UTC m=+433.708467379" Jul 06 19:13:26.195674 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:26.195653 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-blzqc"] Jul 06 19:13:28.137531 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:28.137479 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-blzqc" podUID="071c63bb-2594-44b3-853d-20bd8ff57c77" containerName="authorino" containerID="cri-o://5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85" gracePeriod=30 Jul 06 19:13:28.367815 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:28.367795 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-blzqc" Jul 06 19:13:28.496664 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:28.496605 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgzzj\" (UniqueName: \"kubernetes.io/projected/071c63bb-2594-44b3-853d-20bd8ff57c77-kube-api-access-vgzzj\") pod \"071c63bb-2594-44b3-853d-20bd8ff57c77\" (UID: \"071c63bb-2594-44b3-853d-20bd8ff57c77\") " Jul 06 19:13:28.498460 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:28.498437 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071c63bb-2594-44b3-853d-20bd8ff57c77-kube-api-access-vgzzj" (OuterVolumeSpecName: "kube-api-access-vgzzj") pod "071c63bb-2594-44b3-853d-20bd8ff57c77" (UID: "071c63bb-2594-44b3-853d-20bd8ff57c77"). InnerVolumeSpecName "kube-api-access-vgzzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:13:28.597377 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:28.597350 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgzzj\" (UniqueName: \"kubernetes.io/projected/071c63bb-2594-44b3-853d-20bd8ff57c77-kube-api-access-vgzzj\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:13:29.143974 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.143943 2569 generic.go:356] "Generic (PLEG): container finished" podID="071c63bb-2594-44b3-853d-20bd8ff57c77" containerID="5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85" exitCode=0 Jul 06 19:13:29.144310 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.143990 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-blzqc" Jul 06 19:13:29.144310 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.144029 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-blzqc" event={"ID":"071c63bb-2594-44b3-853d-20bd8ff57c77","Type":"ContainerDied","Data":"5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85"} Jul 06 19:13:29.144310 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.144072 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-blzqc" event={"ID":"071c63bb-2594-44b3-853d-20bd8ff57c77","Type":"ContainerDied","Data":"a7f8b51059d7933be23de3ee11dc5c8c26150249d718e3761b4e833077ce0f8a"} Jul 06 19:13:29.144310 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.144092 2569 scope.go:117] "RemoveContainer" containerID="5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85" Jul 06 19:13:29.153226 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.153200 2569 scope.go:117] "RemoveContainer" containerID="5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85" Jul 06 19:13:29.153517 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:13:29.153492 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85\": container with ID starting with 5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85 not found: ID does not exist" containerID="5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85" Jul 06 19:13:29.153614 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.153524 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85"} err="failed to get container status \"5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85\": rpc error: code = NotFound desc = could not find container \"5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85\": container with ID starting with 5c3a541c7c3150a5baef14f9eeeea481ec1cbc1e03d75f9054a452a264218b85 not found: ID does not exist" Jul 06 19:13:29.161165 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.161145 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-blzqc"] Jul 06 19:13:29.163306 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.163285 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-blzqc"] Jul 06 19:13:29.386692 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.386625 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:29.444291 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:29.444264 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:31.021705 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:31.021664 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="071c63bb-2594-44b3-853d-20bd8ff57c77" path="/var/lib/kubelet/pods/071c63bb-2594-44b3-853d-20bd8ff57c77/volumes" Jul 06 19:13:37.133344 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:37.133268 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-plldc" Jul 06 19:13:41.390732 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:41.390697 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:41.483513 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:41.483475 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:41.852474 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:41.852446 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:53.547953 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.547911 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:53.814700 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.814630 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-25cnd"] Jul 06 19:13:53.814982 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.814968 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="071c63bb-2594-44b3-853d-20bd8ff57c77" containerName="authorino" Jul 06 19:13:53.815026 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.814982 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="071c63bb-2594-44b3-853d-20bd8ff57c77" containerName="authorino" Jul 06 19:13:53.815109 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.815098 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="071c63bb-2594-44b3-853d-20bd8ff57c77" containerName="authorino" Jul 06 19:13:53.817634 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.817615 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-25cnd" Jul 06 19:13:53.824316 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.824291 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-25cnd"] Jul 06 19:13:53.874499 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.874471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m27vr\" (UniqueName: \"kubernetes.io/projected/afc84b96-bb3a-4d44-8539-6c7637ffe122-kube-api-access-m27vr\") pod \"authorino-5d4b67c446-25cnd\" (UID: \"afc84b96-bb3a-4d44-8539-6c7637ffe122\") " pod="kuadrant-system/authorino-5d4b67c446-25cnd" Jul 06 19:13:53.914221 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.914196 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:53.975521 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.975496 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m27vr\" (UniqueName: \"kubernetes.io/projected/afc84b96-bb3a-4d44-8539-6c7637ffe122-kube-api-access-m27vr\") pod \"authorino-5d4b67c446-25cnd\" (UID: \"afc84b96-bb3a-4d44-8539-6c7637ffe122\") " pod="kuadrant-system/authorino-5d4b67c446-25cnd" Jul 06 19:13:53.983730 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:53.983708 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m27vr\" (UniqueName: \"kubernetes.io/projected/afc84b96-bb3a-4d44-8539-6c7637ffe122-kube-api-access-m27vr\") pod \"authorino-5d4b67c446-25cnd\" (UID: \"afc84b96-bb3a-4d44-8539-6c7637ffe122\") " pod="kuadrant-system/authorino-5d4b67c446-25cnd" Jul 06 19:13:54.106671 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.106616 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-25cnd"] Jul 06 19:13:54.106824 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.106809 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-25cnd" Jul 06 19:13:54.142830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.142801 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75884bd857-xkdj7"] Jul 06 19:13:54.146662 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.146644 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:13:54.149203 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.149137 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 19:13:54.152653 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.152633 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75884bd857-xkdj7"] Jul 06 19:13:54.177591 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.177564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-tls-cert\") pod \"authorino-75884bd857-xkdj7\" (UID: \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\") " pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:13:54.177714 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.177622 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkgtn\" (UniqueName: \"kubernetes.io/projected/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-kube-api-access-qkgtn\") pod \"authorino-75884bd857-xkdj7\" (UID: \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\") " pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:13:54.243447 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.243406 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-25cnd"] Jul 06 19:13:54.245223 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:13:54.245192 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafc84b96_bb3a_4d44_8539_6c7637ffe122.slice/crio-8d5e32498be1a6cd119f07b383fba452d1215f40fee066478a23d0fa719b3eb0 WatchSource:0}: Error finding container 8d5e32498be1a6cd119f07b383fba452d1215f40fee066478a23d0fa719b3eb0: Status 404 returned error can't find the container with id 8d5e32498be1a6cd119f07b383fba452d1215f40fee066478a23d0fa719b3eb0 Jul 06 19:13:54.278346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.278323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-tls-cert\") pod \"authorino-75884bd857-xkdj7\" (UID: \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\") " pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:13:54.278479 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.278362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkgtn\" (UniqueName: \"kubernetes.io/projected/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-kube-api-access-qkgtn\") pod \"authorino-75884bd857-xkdj7\" (UID: \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\") " pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:13:54.279360 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.279338 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-25cnd" event={"ID":"afc84b96-bb3a-4d44-8539-6c7637ffe122","Type":"ContainerStarted","Data":"8d5e32498be1a6cd119f07b383fba452d1215f40fee066478a23d0fa719b3eb0"} Jul 06 19:13:54.280822 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.280799 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-tls-cert\") pod \"authorino-75884bd857-xkdj7\" (UID: \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\") " pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:13:54.287952 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.287925 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkgtn\" (UniqueName: \"kubernetes.io/projected/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-kube-api-access-qkgtn\") pod \"authorino-75884bd857-xkdj7\" (UID: \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\") " pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:13:54.459715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.459661 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:13:54.581871 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:54.581843 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75884bd857-xkdj7"] Jul 06 19:13:54.583758 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:13:54.583727 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd23f970_8aa1_4b18_9f89_73acf8aa08c4.slice/crio-dadcbac8e903d5ff49aa9f44beccdf28c7484504e61b55fa5a2d09013d2928d8 WatchSource:0}: Error finding container dadcbac8e903d5ff49aa9f44beccdf28c7484504e61b55fa5a2d09013d2928d8: Status 404 returned error can't find the container with id dadcbac8e903d5ff49aa9f44beccdf28c7484504e61b55fa5a2d09013d2928d8 Jul 06 19:13:55.286687 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.286649 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75884bd857-xkdj7" event={"ID":"cd23f970-8aa1-4b18-9f89-73acf8aa08c4","Type":"ContainerStarted","Data":"9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c"} Jul 06 19:13:55.286865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.286698 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75884bd857-xkdj7" event={"ID":"cd23f970-8aa1-4b18-9f89-73acf8aa08c4","Type":"ContainerStarted","Data":"dadcbac8e903d5ff49aa9f44beccdf28c7484504e61b55fa5a2d09013d2928d8"} Jul 06 19:13:55.288283 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.288246 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-25cnd" event={"ID":"afc84b96-bb3a-4d44-8539-6c7637ffe122","Type":"ContainerStarted","Data":"53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa"} Jul 06 19:13:55.288283 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.288250 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-25cnd" podUID="afc84b96-bb3a-4d44-8539-6c7637ffe122" containerName="authorino" containerID="cri-o://53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa" gracePeriod=30 Jul 06 19:13:55.328286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.328243 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75884bd857-xkdj7" podStartSLOduration=1.011102664 podStartE2EDuration="1.328227507s" totalImagesPullingTime="317.124843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:55.306865238 +0000 UTC m=+462.847587464" watchObservedRunningTime="2026-07-06 19:13:55.328227507 +0000 UTC m=+462.868949748" Jul 06 19:13:55.339796 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.339754 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-25cnd" podStartSLOduration=1.957110656 podStartE2EDuration="2.339740812s" totalImagesPullingTime="382.630156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:55.328025082 +0000 UTC m=+462.868747311" watchObservedRunningTime="2026-07-06 19:13:55.339740812 +0000 UTC m=+462.880463078" Jul 06 19:13:55.341523 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.341500 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tgkxt"] Jul 06 19:13:55.341712 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.341692 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-tgkxt" podUID="3b9efed4-c5d3-4309-91c8-8e8bd992f08d" containerName="authorino" containerID="cri-o://56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995" gracePeriod=30 Jul 06 19:13:55.594156 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.594134 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-25cnd" Jul 06 19:13:55.608109 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.608090 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tgkxt" Jul 06 19:13:55.689460 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.689415 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m27vr\" (UniqueName: \"kubernetes.io/projected/afc84b96-bb3a-4d44-8539-6c7637ffe122-kube-api-access-m27vr\") pod \"afc84b96-bb3a-4d44-8539-6c7637ffe122\" (UID: \"afc84b96-bb3a-4d44-8539-6c7637ffe122\") " Jul 06 19:13:55.689599 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.689469 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b862t\" (UniqueName: \"kubernetes.io/projected/3b9efed4-c5d3-4309-91c8-8e8bd992f08d-kube-api-access-b862t\") pod \"3b9efed4-c5d3-4309-91c8-8e8bd992f08d\" (UID: \"3b9efed4-c5d3-4309-91c8-8e8bd992f08d\") " Jul 06 19:13:55.691436 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.691400 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9efed4-c5d3-4309-91c8-8e8bd992f08d-kube-api-access-b862t" (OuterVolumeSpecName: "kube-api-access-b862t") pod "3b9efed4-c5d3-4309-91c8-8e8bd992f08d" (UID: "3b9efed4-c5d3-4309-91c8-8e8bd992f08d"). InnerVolumeSpecName "kube-api-access-b862t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:13:55.691535 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.691454 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc84b96-bb3a-4d44-8539-6c7637ffe122-kube-api-access-m27vr" (OuterVolumeSpecName: "kube-api-access-m27vr") pod "afc84b96-bb3a-4d44-8539-6c7637ffe122" (UID: "afc84b96-bb3a-4d44-8539-6c7637ffe122"). InnerVolumeSpecName "kube-api-access-m27vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:13:55.790845 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.790821 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m27vr\" (UniqueName: \"kubernetes.io/projected/afc84b96-bb3a-4d44-8539-6c7637ffe122-kube-api-access-m27vr\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:13:55.790845 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:55.790840 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b862t\" (UniqueName: \"kubernetes.io/projected/3b9efed4-c5d3-4309-91c8-8e8bd992f08d-kube-api-access-b862t\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:13:56.294338 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.294310 2569 generic.go:356] "Generic (PLEG): container finished" podID="3b9efed4-c5d3-4309-91c8-8e8bd992f08d" containerID="56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995" exitCode=0 Jul 06 19:13:56.294513 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.294347 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tgkxt" Jul 06 19:13:56.294513 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.294390 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tgkxt" event={"ID":"3b9efed4-c5d3-4309-91c8-8e8bd992f08d","Type":"ContainerDied","Data":"56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995"} Jul 06 19:13:56.294513 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.294436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tgkxt" event={"ID":"3b9efed4-c5d3-4309-91c8-8e8bd992f08d","Type":"ContainerDied","Data":"ed2192948a3ffaf68acbdf88d97b7f399530e7282732a70159fb2790416c147a"} Jul 06 19:13:56.294513 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.294455 2569 scope.go:117] "RemoveContainer" containerID="56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995" Jul 06 19:13:56.295838 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.295814 2569 generic.go:356] "Generic (PLEG): container finished" podID="afc84b96-bb3a-4d44-8539-6c7637ffe122" containerID="53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa" exitCode=0 Jul 06 19:13:56.295954 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.295871 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-25cnd" Jul 06 19:13:56.295954 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.295903 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-25cnd" event={"ID":"afc84b96-bb3a-4d44-8539-6c7637ffe122","Type":"ContainerDied","Data":"53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa"} Jul 06 19:13:56.295954 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.295941 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-25cnd" event={"ID":"afc84b96-bb3a-4d44-8539-6c7637ffe122","Type":"ContainerDied","Data":"8d5e32498be1a6cd119f07b383fba452d1215f40fee066478a23d0fa719b3eb0"} Jul 06 19:13:56.304561 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.304540 2569 scope.go:117] "RemoveContainer" containerID="56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995" Jul 06 19:13:56.304900 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:13:56.304832 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995\": container with ID starting with 56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995 not found: ID does not exist" containerID="56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995" Jul 06 19:13:56.304900 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.304858 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995"} err="failed to get container status \"56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995\": rpc error: code = NotFound desc = could not find container \"56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995\": container with ID starting with 56813d3e221a799b45dfd0042fb7963f61d4e43b3993fc7985084155dcc3e995 not found: ID does not exist" Jul 06 19:13:56.304900 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.304876 2569 scope.go:117] "RemoveContainer" containerID="53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa" Jul 06 19:13:56.312304 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.312288 2569 scope.go:117] "RemoveContainer" containerID="53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa" Jul 06 19:13:56.312565 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:13:56.312547 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa\": container with ID starting with 53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa not found: ID does not exist" containerID="53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa" Jul 06 19:13:56.312636 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.312574 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa"} err="failed to get container status \"53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa\": rpc error: code = NotFound desc = could not find container \"53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa\": container with ID starting with 53c685847cd1743ce8aa842037ef3473fa3ace736e90d4cd54b8b39227c886aa not found: ID does not exist" Jul 06 19:13:56.321865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.321846 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-25cnd"] Jul 06 19:13:56.324749 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.324728 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-25cnd"] Jul 06 19:13:56.327579 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.327562 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:56.345360 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.345338 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tgkxt"] Jul 06 19:13:56.347196 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.347179 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tgkxt"] Jul 06 19:13:56.396762 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:56.396740 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:13:57.021792 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:57.021762 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9efed4-c5d3-4309-91c8-8e8bd992f08d" path="/var/lib/kubelet/pods/3b9efed4-c5d3-4309-91c8-8e8bd992f08d/volumes" Jul 06 19:13:57.022148 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:57.022051 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afc84b96-bb3a-4d44-8539-6c7637ffe122" path="/var/lib/kubelet/pods/afc84b96-bb3a-4d44-8539-6c7637ffe122/volumes" Jul 06 19:13:59.969905 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:13:59.969873 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:00.065257 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:00.065230 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:00.161478 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:00.161449 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:00.368021 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:00.367984 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:03.979181 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:03.979147 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:04.141113 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:04.141076 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:27.888749 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:27.888703 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:35.460958 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:35.460923 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:36.029239 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:36.029207 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:36.574483 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:36.574453 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:36.676878 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:36.676836 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:36.769368 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:36.769339 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:14:46.913152 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:46.913124 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:14:47.003313 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:14:47.003284 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:02.290036 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:02.290003 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:02.676136 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:02.676049 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:15:09.555389 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:09.555314 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:18.248691 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.248663 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss"] Jul 06 19:15:18.249098 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.248998 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc84b96-bb3a-4d44-8539-6c7637ffe122" containerName="authorino" Jul 06 19:15:18.249098 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.249008 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc84b96-bb3a-4d44-8539-6c7637ffe122" containerName="authorino" Jul 06 19:15:18.249098 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.249020 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9efed4-c5d3-4309-91c8-8e8bd992f08d" containerName="authorino" Jul 06 19:15:18.249098 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.249025 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9efed4-c5d3-4309-91c8-8e8bd992f08d" containerName="authorino" Jul 06 19:15:18.249222 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.249106 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9efed4-c5d3-4309-91c8-8e8bd992f08d" containerName="authorino" Jul 06 19:15:18.249222 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.249113 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc84b96-bb3a-4d44-8539-6c7637ffe122" containerName="authorino" Jul 06 19:15:18.251256 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.251241 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.253524 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.253504 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:15:18.253644 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.253529 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 19:15:18.254327 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.254311 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:15:18.254327 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.254317 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jxxj4\"" Jul 06 19:15:18.261263 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.261246 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss"] Jul 06 19:15:18.275849 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.275824 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:18.290045 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.290023 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.290140 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.290053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.290201 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.290136 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.290201 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.290162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.290300 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.290211 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9qs\" (UniqueName: \"kubernetes.io/projected/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-kube-api-access-xk9qs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.290300 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.290248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.390560 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.390538 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.390666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.390571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.390666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.390628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.390666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.390651 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.390814 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.390769 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9qs\" (UniqueName: \"kubernetes.io/projected/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-kube-api-access-xk9qs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.390874 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.390814 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.391038 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.391016 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.391089 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.391035 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.391171 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.391152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.392761 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.392742 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.393479 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.393458 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.399188 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.399169 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9qs\" (UniqueName: \"kubernetes.io/projected/d6d71f96-0ae5-4826-9fe8-c5117bed1e81-kube-api-access-xk9qs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v77ss\" (UID: \"d6d71f96-0ae5-4826-9fe8-c5117bed1e81\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.562790 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.562765 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:18.687630 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.687608 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss"] Jul 06 19:15:18.689584 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:15:18.689551 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d71f96_0ae5_4826_9fe8_c5117bed1e81.slice/crio-7c4418bcc7fc352e7a55ec6f8f96b8a22fcda748a11c6a5b502c0aa155ad78b1 WatchSource:0}: Error finding container 7c4418bcc7fc352e7a55ec6f8f96b8a22fcda748a11c6a5b502c0aa155ad78b1: Status 404 returned error can't find the container with id 7c4418bcc7fc352e7a55ec6f8f96b8a22fcda748a11c6a5b502c0aa155ad78b1 Jul 06 19:15:18.745594 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:18.745565 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" event={"ID":"d6d71f96-0ae5-4826-9fe8-c5117bed1e81","Type":"ContainerStarted","Data":"7c4418bcc7fc352e7a55ec6f8f96b8a22fcda748a11c6a5b502c0aa155ad78b1"} Jul 06 19:15:19.060386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:19.060352 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:15:20.757692 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:20.757660 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" event={"ID":"d6d71f96-0ae5-4826-9fe8-c5117bed1e81","Type":"ContainerStarted","Data":"83d59047201ddc91f0803a4d32f45d6a69bd4330d21ad897877a9b672f72b720"} Jul 06 19:15:20.778837 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:20.778795 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" podStartSLOduration=1.704040013 podStartE2EDuration="2.778781591s" totalImagesPullingTime="1.074741578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:20.776763026 +0000 UTC m=+548.317485254" watchObservedRunningTime="2026-07-06 19:15:20.778781591 +0000 UTC m=+548.319503819" Jul 06 19:15:28.563029 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:28.562995 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:28.563387 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:28.563060 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:28.575058 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:28.575039 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:28.812203 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:28.812182 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v77ss" Jul 06 19:15:31.032112 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:31.032083 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:32.053214 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:32.053182 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:15:36.575899 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:36.575863 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:37.960021 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:37.959984 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:15:55.857498 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.857458 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs"] Jul 06 19:15:55.860968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.860952 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:55.863904 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.863888 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 19:15:55.871163 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.871142 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs"] Jul 06 19:15:55.967617 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.967585 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:15:55.967767 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.967748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7tpc\" (UniqueName: \"kubernetes.io/projected/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-kube-api-access-j7tpc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:55.967848 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.967778 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:55.967848 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.967795 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:55.967848 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.967811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:55.967945 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.967897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:55.967945 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:55.967920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.068981 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.068955 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7tpc\" (UniqueName: \"kubernetes.io/projected/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-kube-api-access-j7tpc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.068981 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.068982 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.069141 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.069000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.069141 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.069016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.069141 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.069037 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.069141 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.069051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.069393 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.069376 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.069530 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.069512 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.069612 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.069591 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.071372 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.071353 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.071459 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.071410 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.076632 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.076607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7tpc\" (UniqueName: \"kubernetes.io/projected/95581bfd-0ca3-4c28-b8f2-b677e2159ab0-kube-api-access-j7tpc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2vwrs\" (UID: \"95581bfd-0ca3-4c28-b8f2-b677e2159ab0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.171011 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.170956 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:15:56.299124 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.299097 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs"] Jul 06 19:15:56.301195 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:15:56.301169 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95581bfd_0ca3_4c28_b8f2_b677e2159ab0.slice/crio-0583d1ecce5d5279f91c76500a8d3356ad92b968e905ef86e38c07dfc171202c WatchSource:0}: Error finding container 0583d1ecce5d5279f91c76500a8d3356ad92b968e905ef86e38c07dfc171202c: Status 404 returned error can't find the container with id 0583d1ecce5d5279f91c76500a8d3356ad92b968e905ef86e38c07dfc171202c Jul 06 19:15:56.948488 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.948388 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" event={"ID":"95581bfd-0ca3-4c28-b8f2-b677e2159ab0","Type":"ContainerStarted","Data":"c60a6deb6257c82bc78c2429d4a6ed6613529079cd8eb570ee12067729076dc4"} Jul 06 19:15:56.948488 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.948458 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" event={"ID":"95581bfd-0ca3-4c28-b8f2-b677e2159ab0","Type":"ContainerStarted","Data":"0583d1ecce5d5279f91c76500a8d3356ad92b968e905ef86e38c07dfc171202c"} Jul 06 19:15:56.973123 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:56.973079 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" podStartSLOduration=1.7507794300000001 podStartE2EDuration="1.973063706s" totalImagesPullingTime="222.284276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:56.970703713 +0000 UTC m=+584.511425940" watchObservedRunningTime="2026-07-06 19:15:56.973063706 +0000 UTC m=+584.513785933" Jul 06 19:15:57.996275 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:15:57.996239 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:16:02.064418 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.064388 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.164183 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.164151 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.851768 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.851739 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5"] Jul 06 19:16:02.855471 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.855448 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:02.857819 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.857786 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 19:16:02.865349 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.865329 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5"] Jul 06 19:16:02.895158 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.895121 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:02.922373 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.922351 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:02.922500 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.922381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:02.922500 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.922488 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27f6\" (UniqueName: \"kubernetes.io/projected/919ccf05-a449-40a8-9f44-c67a6a3214d6-kube-api-access-t27f6\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:02.922579 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.922520 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919ccf05-a449-40a8-9f44-c67a6a3214d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:02.922579 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.922556 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:02.922641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:02.922579 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.023601 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.023578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27f6\" (UniqueName: \"kubernetes.io/projected/919ccf05-a449-40a8-9f44-c67a6a3214d6-kube-api-access-t27f6\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.023741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.023612 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919ccf05-a449-40a8-9f44-c67a6a3214d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.023741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.023637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.023741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.023651 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.023741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.023683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.023741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.023705 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.023994 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.023953 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.024062 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.024018 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.024163 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.024143 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.025808 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.025788 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919ccf05-a449-40a8-9f44-c67a6a3214d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.025961 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.025945 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919ccf05-a449-40a8-9f44-c67a6a3214d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.031239 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.031220 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27f6\" (UniqueName: \"kubernetes.io/projected/919ccf05-a449-40a8-9f44-c67a6a3214d6-kube-api-access-t27f6\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5\" (UID: \"919ccf05-a449-40a8-9f44-c67a6a3214d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.165805 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.165749 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:03.290197 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.290176 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5"] Jul 06 19:16:03.292016 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:16:03.291979 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod919ccf05_a449_40a8_9f44_c67a6a3214d6.slice/crio-c10535f6b6243cb8e2653b07cc5e56351c20c7ffa474da36e7cd85410f050dcf WatchSource:0}: Error finding container c10535f6b6243cb8e2653b07cc5e56351c20c7ffa474da36e7cd85410f050dcf: Status 404 returned error can't find the container with id c10535f6b6243cb8e2653b07cc5e56351c20c7ffa474da36e7cd85410f050dcf Jul 06 19:16:03.989795 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.989762 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" event={"ID":"919ccf05-a449-40a8-9f44-c67a6a3214d6","Type":"ContainerStarted","Data":"9a9772cddaa36cb368178ccf61dfe4f57d0513ad16d418c131025f0535082b12"} Jul 06 19:16:03.989795 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:03.989802 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" event={"ID":"919ccf05-a449-40a8-9f44-c67a6a3214d6","Type":"ContainerStarted","Data":"c10535f6b6243cb8e2653b07cc5e56351c20c7ffa474da36e7cd85410f050dcf"} Jul 06 19:16:04.009191 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:04.009146 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" podStartSLOduration=1.839625764 podStartE2EDuration="2.00913181s" totalImagesPullingTime="169.506046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:04.008298286 +0000 UTC m=+591.549020513" watchObservedRunningTime="2026-07-06 19:16:04.00913181 +0000 UTC m=+591.549854070" Jul 06 19:16:06.171972 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:06.171932 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:16:06.172411 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:06.172005 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:16:06.183963 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:06.183940 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:16:06.777850 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:06.777817 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:16:07.021870 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:07.021842 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2vwrs" Jul 06 19:16:12.890254 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:12.890225 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:16:12.890762 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:12.890271 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:16:13.165952 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:13.165877 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:13.165952 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:13.165919 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:13.177749 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:13.177730 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:14.053270 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:14.053241 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5" Jul 06 19:16:38.612853 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:38.612747 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:38.677022 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:38.676994 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:39.602982 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:39.602950 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:39.662938 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:39.662898 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:56.104071 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.104032 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-664684f5d4-w9zrl"] Jul 06 19:16:56.107626 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.107609 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-664684f5d4-w9zrl" Jul 06 19:16:56.117983 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.117959 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-664684f5d4-w9zrl"] Jul 06 19:16:56.148508 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.148470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qrm\" (UniqueName: \"kubernetes.io/projected/ece53de8-6b72-4c33-83f4-8dccd52e1c12-kube-api-access-t2qrm\") pod \"authorino-664684f5d4-w9zrl\" (UID: \"ece53de8-6b72-4c33-83f4-8dccd52e1c12\") " pod="kuadrant-system/authorino-664684f5d4-w9zrl" Jul 06 19:16:56.148508 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.148506 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ece53de8-6b72-4c33-83f4-8dccd52e1c12-tls-cert\") pod \"authorino-664684f5d4-w9zrl\" (UID: \"ece53de8-6b72-4c33-83f4-8dccd52e1c12\") " pod="kuadrant-system/authorino-664684f5d4-w9zrl" Jul 06 19:16:56.249254 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.249217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qrm\" (UniqueName: \"kubernetes.io/projected/ece53de8-6b72-4c33-83f4-8dccd52e1c12-kube-api-access-t2qrm\") pod \"authorino-664684f5d4-w9zrl\" (UID: \"ece53de8-6b72-4c33-83f4-8dccd52e1c12\") " pod="kuadrant-system/authorino-664684f5d4-w9zrl" Jul 06 19:16:56.249254 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.249253 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ece53de8-6b72-4c33-83f4-8dccd52e1c12-tls-cert\") pod \"authorino-664684f5d4-w9zrl\" (UID: \"ece53de8-6b72-4c33-83f4-8dccd52e1c12\") " pod="kuadrant-system/authorino-664684f5d4-w9zrl" Jul 06 19:16:56.251758 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.251738 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ece53de8-6b72-4c33-83f4-8dccd52e1c12-tls-cert\") pod \"authorino-664684f5d4-w9zrl\" (UID: \"ece53de8-6b72-4c33-83f4-8dccd52e1c12\") " pod="kuadrant-system/authorino-664684f5d4-w9zrl" Jul 06 19:16:56.258455 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.258393 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qrm\" (UniqueName: \"kubernetes.io/projected/ece53de8-6b72-4c33-83f4-8dccd52e1c12-kube-api-access-t2qrm\") pod \"authorino-664684f5d4-w9zrl\" (UID: \"ece53de8-6b72-4c33-83f4-8dccd52e1c12\") " pod="kuadrant-system/authorino-664684f5d4-w9zrl" Jul 06 19:16:56.417613 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.417524 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-664684f5d4-w9zrl" Jul 06 19:16:56.594079 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:16:56.594046 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podece53de8_6b72_4c33_83f4_8dccd52e1c12.slice/crio-df205265840a28e50252f0ff759810c1bb6401b1d3b45ff53b59e34b0f90ebd7 WatchSource:0}: Error finding container df205265840a28e50252f0ff759810c1bb6401b1d3b45ff53b59e34b0f90ebd7: Status 404 returned error can't find the container with id df205265840a28e50252f0ff759810c1bb6401b1d3b45ff53b59e34b0f90ebd7 Jul 06 19:16:56.596007 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.595986 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:16:56.596626 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.596564 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-664684f5d4-w9zrl"] Jul 06 19:16:56.657552 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.657518 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:56.774086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:56.774057 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:57.268785 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.268740 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-664684f5d4-w9zrl" event={"ID":"ece53de8-6b72-4c33-83f4-8dccd52e1c12","Type":"ContainerStarted","Data":"a5964ee97f2ccdb7ce6537a5170af145bb5c2e7102ca6a4b63538caa208e1547"} Jul 06 19:16:57.269209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.268799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-664684f5d4-w9zrl" event={"ID":"ece53de8-6b72-4c33-83f4-8dccd52e1c12","Type":"ContainerStarted","Data":"df205265840a28e50252f0ff759810c1bb6401b1d3b45ff53b59e34b0f90ebd7"} Jul 06 19:16:57.290979 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.290928 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-664684f5d4-w9zrl" podStartSLOduration=0.966740064 podStartE2EDuration="1.290910509s" totalImagesPullingTime="324.170445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:57.288420349 +0000 UTC m=+644.829142579" watchObservedRunningTime="2026-07-06 19:16:57.290910509 +0000 UTC m=+644.831632737" Jul 06 19:16:57.333771 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.333734 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75884bd857-xkdj7"] Jul 06 19:16:57.333972 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.333944 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75884bd857-xkdj7" podUID="cd23f970-8aa1-4b18-9f89-73acf8aa08c4" containerName="authorino" containerID="cri-o://9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c" gracePeriod=30 Jul 06 19:16:57.598890 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.598863 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:16:57.660455 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.660398 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-tls-cert\") pod \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\" (UID: \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\") " Jul 06 19:16:57.660637 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.660487 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkgtn\" (UniqueName: \"kubernetes.io/projected/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-kube-api-access-qkgtn\") pod \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\" (UID: \"cd23f970-8aa1-4b18-9f89-73acf8aa08c4\") " Jul 06 19:16:57.662530 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.662494 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-kube-api-access-qkgtn" (OuterVolumeSpecName: "kube-api-access-qkgtn") pod "cd23f970-8aa1-4b18-9f89-73acf8aa08c4" (UID: "cd23f970-8aa1-4b18-9f89-73acf8aa08c4"). InnerVolumeSpecName "kube-api-access-qkgtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:16:57.672010 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.671980 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cd23f970-8aa1-4b18-9f89-73acf8aa08c4" (UID: "cd23f970-8aa1-4b18-9f89-73acf8aa08c4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:16:57.751057 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.751015 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:57.761380 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.761353 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkgtn\" (UniqueName: \"kubernetes.io/projected/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-kube-api-access-qkgtn\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:16:57.761380 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.761376 2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cd23f970-8aa1-4b18-9f89-73acf8aa08c4-tls-cert\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:16:57.855017 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:57.854913 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:16:58.284336 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.284303 2569 generic.go:356] "Generic (PLEG): container finished" podID="cd23f970-8aa1-4b18-9f89-73acf8aa08c4" containerID="9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c" exitCode=0 Jul 06 19:16:58.284722 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.284375 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75884bd857-xkdj7" Jul 06 19:16:58.284722 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.284391 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75884bd857-xkdj7" event={"ID":"cd23f970-8aa1-4b18-9f89-73acf8aa08c4","Type":"ContainerDied","Data":"9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c"} Jul 06 19:16:58.284722 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.284446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75884bd857-xkdj7" event={"ID":"cd23f970-8aa1-4b18-9f89-73acf8aa08c4","Type":"ContainerDied","Data":"dadcbac8e903d5ff49aa9f44beccdf28c7484504e61b55fa5a2d09013d2928d8"} Jul 06 19:16:58.284722 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.284532 2569 scope.go:117] "RemoveContainer" containerID="9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c" Jul 06 19:16:58.295309 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.295288 2569 scope.go:117] "RemoveContainer" containerID="9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c" Jul 06 19:16:58.295727 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:16:58.295674 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c\": container with ID starting with 9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c not found: ID does not exist" containerID="9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c" Jul 06 19:16:58.295844 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.295739 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c"} err="failed to get container status \"9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c\": rpc error: code = NotFound desc = could not find container \"9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c\": container with ID starting with 9e18f87ac54f5a69afa3f0b701d83957141a99f4e81741f57852b5502f628e3c not found: ID does not exist" Jul 06 19:16:58.307017 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.306995 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75884bd857-xkdj7"] Jul 06 19:16:58.311017 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:58.310996 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75884bd857-xkdj7"] Jul 06 19:16:59.022388 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:16:59.022352 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd23f970-8aa1-4b18-9f89-73acf8aa08c4" path="/var/lib/kubelet/pods/cd23f970-8aa1-4b18-9f89-73acf8aa08c4/volumes" Jul 06 19:17:23.301988 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:17:23.301951 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:24.570985 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:17:24.570944 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:17:35.955941 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:17:35.955905 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:17:46.916691 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:17:46.916659 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:17:47.350280 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:17:47.350250 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:17:56.066255 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:17:56.066216 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:17:57.622025 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:17:57.621990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:06.555635 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:06.555553 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:18:08.886016 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:08.885987 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:10.139385 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:10.139350 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:43.675733 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:43.675677 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:43.771768 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:43.771737 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:43.909839 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:43.909808 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:49.931468 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:49.931407 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:50.006732 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:50.006699 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:50.097363 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:50.097332 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:50.287534 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:50.287502 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:56.445124 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:56.445086 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:18:56.544959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:18:56.544923 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:19:13.153362 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:19:13.153325 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:19:22.952171 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:19:22.952135 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:19:32.680951 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:19:32.680906 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:19:43.750448 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:19:43.750343 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:19:54.365064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:19:54.365026 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:20:03.056318 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:20:03.056286 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:20:15.348177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:20:15.348144 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:20:24.558574 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:20:24.558542 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:20:38.705036 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:20:38.704983 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:20:47.354712 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:20:47.354681 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:20:57.053934 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:20:57.053905 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:21:12.926726 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:21:12.926652 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:21:12.929299 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:21:12.929266 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:21:19.555182 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:21:19.555146 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:21:31.354847 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:21:31.354810 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:21:47.653920 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:21:47.653885 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:21:57.057787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:21:57.057748 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:22:06.254926 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:06.254884 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:22:15.451494 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:15.451458 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:22:23.719736 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:23.719696 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:22:24.360536 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:24.360497 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:22:51.216476 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:51.216363 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz"] Jul 06 19:22:51.217069 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:51.216584 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" podUID="67b91cd9-892a-4e61-9a3c-a23e09926a07" containerName="manager" containerID="cri-o://3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17" gracePeriod=10 Jul 06 19:22:54.164244 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.164221 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:22:54.206382 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.206348 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b91cd9-892a-4e61-9a3c-a23e09926a07-extensions-socket-volume\") pod \"67b91cd9-892a-4e61-9a3c-a23e09926a07\" (UID: \"67b91cd9-892a-4e61-9a3c-a23e09926a07\") " Jul 06 19:22:54.206561 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.206438 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vllgq\" (UniqueName: \"kubernetes.io/projected/67b91cd9-892a-4e61-9a3c-a23e09926a07-kube-api-access-vllgq\") pod \"67b91cd9-892a-4e61-9a3c-a23e09926a07\" (UID: \"67b91cd9-892a-4e61-9a3c-a23e09926a07\") " Jul 06 19:22:54.206749 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.206727 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b91cd9-892a-4e61-9a3c-a23e09926a07-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "67b91cd9-892a-4e61-9a3c-a23e09926a07" (UID: "67b91cd9-892a-4e61-9a3c-a23e09926a07"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:22:54.208555 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.208532 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b91cd9-892a-4e61-9a3c-a23e09926a07-kube-api-access-vllgq" (OuterVolumeSpecName: "kube-api-access-vllgq") pod "67b91cd9-892a-4e61-9a3c-a23e09926a07" (UID: "67b91cd9-892a-4e61-9a3c-a23e09926a07"). InnerVolumeSpecName "kube-api-access-vllgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:22:54.261177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.261082 2569 generic.go:356] "Generic (PLEG): container finished" podID="67b91cd9-892a-4e61-9a3c-a23e09926a07" containerID="3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17" exitCode=0 Jul 06 19:22:54.261177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.261118 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" Jul 06 19:22:54.261177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.261116 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" event={"ID":"67b91cd9-892a-4e61-9a3c-a23e09926a07","Type":"ContainerDied","Data":"3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17"} Jul 06 19:22:54.261177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.261169 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz" event={"ID":"67b91cd9-892a-4e61-9a3c-a23e09926a07","Type":"ContainerDied","Data":"a420f391149a97aebe07191cc987141a323ca5197f8e6f8d516e8dca7e5b3b78"} Jul 06 19:22:54.261481 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.261191 2569 scope.go:117] "RemoveContainer" containerID="3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17" Jul 06 19:22:54.271494 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.271478 2569 scope.go:117] "RemoveContainer" containerID="3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17" Jul 06 19:22:54.271744 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:22:54.271726 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17\": container with ID starting with 3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17 not found: ID does not exist" containerID="3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17" Jul 06 19:22:54.271812 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.271751 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17"} err="failed to get container status \"3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17\": rpc error: code = NotFound desc = could not find container \"3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17\": container with ID starting with 3f07bfe37ff2a3b9873556e786da4bde291766544b18c90002d3e9e18d6e6f17 not found: ID does not exist" Jul 06 19:22:54.283071 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.283044 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz"] Jul 06 19:22:54.288051 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.288029 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pgrlz"] Jul 06 19:22:54.307576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.307553 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b91cd9-892a-4e61-9a3c-a23e09926a07-extensions-socket-volume\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:22:54.307694 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:54.307578 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllgq\" (UniqueName: \"kubernetes.io/projected/67b91cd9-892a-4e61-9a3c-a23e09926a07-kube-api-access-vllgq\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:22:55.021768 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:22:55.021729 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b91cd9-892a-4e61-9a3c-a23e09926a07" path="/var/lib/kubelet/pods/67b91cd9-892a-4e61-9a3c-a23e09926a07/volumes" Jul 06 19:23:52.353801 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.353762 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q"] Jul 06 19:23:52.354261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.354125 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd23f970-8aa1-4b18-9f89-73acf8aa08c4" containerName="authorino" Jul 06 19:23:52.354261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.354135 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd23f970-8aa1-4b18-9f89-73acf8aa08c4" containerName="authorino" Jul 06 19:23:52.354261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.354161 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b91cd9-892a-4e61-9a3c-a23e09926a07" containerName="manager" Jul 06 19:23:52.354261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.354166 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b91cd9-892a-4e61-9a3c-a23e09926a07" containerName="manager" Jul 06 19:23:52.354261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.354222 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b91cd9-892a-4e61-9a3c-a23e09926a07" containerName="manager" Jul 06 19:23:52.354261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.354231 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd23f970-8aa1-4b18-9f89-73acf8aa08c4" containerName="authorino" Jul 06 19:23:52.357115 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.357100 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:52.361833 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.361816 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vpnh8\"" Jul 06 19:23:52.370753 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.370730 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q"] Jul 06 19:23:52.507155 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.507120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjthz\" (UniqueName: \"kubernetes.io/projected/74fd2182-07f3-4075-a9cc-76c6080d9fa7-kube-api-access-kjthz\") pod \"kuadrant-operator-controller-manager-86878c769f-djd9q\" (UID: \"74fd2182-07f3-4075-a9cc-76c6080d9fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:52.507155 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.507162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74fd2182-07f3-4075-a9cc-76c6080d9fa7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-djd9q\" (UID: \"74fd2182-07f3-4075-a9cc-76c6080d9fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:52.607702 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.607607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjthz\" (UniqueName: \"kubernetes.io/projected/74fd2182-07f3-4075-a9cc-76c6080d9fa7-kube-api-access-kjthz\") pod \"kuadrant-operator-controller-manager-86878c769f-djd9q\" (UID: \"74fd2182-07f3-4075-a9cc-76c6080d9fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:52.607702 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.607656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74fd2182-07f3-4075-a9cc-76c6080d9fa7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-djd9q\" (UID: \"74fd2182-07f3-4075-a9cc-76c6080d9fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:52.608056 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.608037 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74fd2182-07f3-4075-a9cc-76c6080d9fa7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-djd9q\" (UID: \"74fd2182-07f3-4075-a9cc-76c6080d9fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:52.615897 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.615876 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjthz\" (UniqueName: \"kubernetes.io/projected/74fd2182-07f3-4075-a9cc-76c6080d9fa7-kube-api-access-kjthz\") pod \"kuadrant-operator-controller-manager-86878c769f-djd9q\" (UID: \"74fd2182-07f3-4075-a9cc-76c6080d9fa7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:52.667621 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.667584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:52.811507 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:52.811485 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q"] Jul 06 19:23:52.813386 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:23:52.813360 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fd2182_07f3_4075_a9cc_76c6080d9fa7.slice/crio-30b0be808e144295a0ddd2229860363b765744a2e6effc22ab9983604e3d266b WatchSource:0}: Error finding container 30b0be808e144295a0ddd2229860363b765744a2e6effc22ab9983604e3d266b: Status 404 returned error can't find the container with id 30b0be808e144295a0ddd2229860363b765744a2e6effc22ab9983604e3d266b Jul 06 19:23:53.576749 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:53.576710 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" event={"ID":"74fd2182-07f3-4075-a9cc-76c6080d9fa7","Type":"ContainerStarted","Data":"fcd1356e34a934a30db722d00f68f8862e55d5f98b758b29cf58f772528a28a1"} Jul 06 19:23:53.577153 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:53.576771 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" event={"ID":"74fd2182-07f3-4075-a9cc-76c6080d9fa7","Type":"ContainerStarted","Data":"30b0be808e144295a0ddd2229860363b765744a2e6effc22ab9983604e3d266b"} Jul 06 19:23:53.577153 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:53.576798 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:23:53.598061 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:23:53.598014 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" podStartSLOduration=1.59799799 podStartE2EDuration="1.59799799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:53.595720859 +0000 UTC m=+1061.136443089" watchObservedRunningTime="2026-07-06 19:23:53.59799799 +0000 UTC m=+1061.138720219" Jul 06 19:24:04.584884 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:04.584849 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-djd9q" Jul 06 19:24:12.257140 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:12.257105 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:24:12.494562 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:12.494528 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:12.581420 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:12.581387 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:16.344645 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:16.344611 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:24:23.156795 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:23.156760 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:24:34.447199 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:34.447168 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:24:42.954172 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:42.954136 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:24:52.752502 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:24:52.752467 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:25:00.752813 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:25:00.752775 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:25:10.561301 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:25:10.561269 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:25:21.150909 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:25:21.150875 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:25:30.757799 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:25:30.757765 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:25:41.050171 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:25:41.050082 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:25:53.151633 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:25:53.151598 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:26:00.158705 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:26:00.158671 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:26:12.960110 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:26:12.960078 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:26:12.964211 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:26:12.964185 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:26:18.571355 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:26:18.571319 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:26:27.754744 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:26:27.754708 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:26:37.068180 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:26:37.068142 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:26:48.649690 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:26:48.649650 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:27:04.158820 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:27:04.158784 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:27:13.654516 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:27:13.654440 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:27:22.484234 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:27:22.484198 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:27:32.458441 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:27:32.458392 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:27:41.365217 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:27:41.365181 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:27:50.549803 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:27:50.549758 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:28:03.254944 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:28:03.254914 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:28:12.558059 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:28:12.558016 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:28:21.952362 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:28:21.952325 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:28:23.268082 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:28:23.268050 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:24.589391 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:28:24.589358 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:33.580498 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:28:33.580463 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:28:45.554660 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:28:45.554586 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:28:54.158694 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:28:54.158656 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:29:05.857526 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:05.857466 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:29:12.846708 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:12.846670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:12.948093 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:12.948056 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:17.751812 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:17.751766 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:29:28.053974 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:28.053934 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:29:30.373205 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:30.373169 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:35.694338 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:35.694282 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:40.252060 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:40.252024 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:29:53.962312 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:53.962277 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:56.854144 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:56.854105 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:29:59.053210 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:29:59.053176 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:08.954476 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:08.954398 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:30:11.856848 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:11.856793 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:11.955861 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:11.955829 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:12.309898 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:12.309862 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:18.321934 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:18.321896 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:18.390742 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:18.390712 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:18.558551 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:18.558517 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:18.666886 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:18.666792 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:19.680892 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:19.680858 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:19.782065 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:19.782035 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:19.894035 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:19.893937 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:19.996986 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:19.996959 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:20.253702 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:20.253630 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:20.355137 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:20.355098 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:22.159639 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:22.159601 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:30:22.277224 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:22.277189 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:24.770719 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:24.770683 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:24.971048 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:24.971015 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:25.063768 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:25.063685 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:25.205082 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:25.205046 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:27.705041 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:27.705005 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:30.353684 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:30.353628 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:30.756477 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:30.756386 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:30:31.229494 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:31.229448 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:31.314903 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:31.314863 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:31.485608 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:31.485514 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:31.554162 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:31.554125 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:32.634177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:32.634140 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:32.735600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:32.735569 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:32.801771 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:32.801728 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:33.169522 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:33.169468 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:33.331187 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:33.331156 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:33.430703 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:33.430620 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:34.648817 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:34.648787 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:37.713414 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:37.713378 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:38.136371 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:38.136334 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:43.783038 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:43.782997 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:43.883561 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:43.883533 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:44.666259 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:44.666225 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:49.940614 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:49.940581 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:50.045887 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:50.045854 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:50.144491 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:50.144449 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:51.251897 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:51.251816 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:51.396302 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:51.396269 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:51.497103 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:51.497071 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:51.915983 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:51.915931 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:52.016879 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:52.016836 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:52.117877 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:52.117837 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:52.190786 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:52.190687 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:53.855175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:53.855143 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:30:55.372759 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:55.372719 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:55.660097 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:55.659955 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:55.871699 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:55.871653 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:56.063220 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:56.063173 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:56.163077 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:56.163045 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:57.638062 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:30:57.638034 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:00.272452 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:00.272398 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:02.363068 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:02.363035 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:31:02.388303 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:02.388272 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:02.489630 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:02.489597 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:02.860496 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:02.860462 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:02.961365 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:02.961332 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:03.060777 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:03.060744 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:04.430229 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:04.430196 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:10.456141 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:10.456102 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:10.564763 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:10.564725 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:10.656259 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:10.656231 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:10.756920 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:10.756840 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:11.165485 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:11.165442 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:12.993747 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:12.993695 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:31:13.000084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:13.000058 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:31:37.161505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:37.161403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:43.094142 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:43.094101 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:43.197471 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:43.197408 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:43.327047 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:43.327012 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:59.177419 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:31:59.177372 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:05.072199 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:05.072170 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:05.176750 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:05.176718 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:05.273826 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:05.273774 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:15.811610 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:15.811578 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:16.640412 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:16.640377 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:23.046700 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:23.046661 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:23.110328 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:23.110297 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:23.193281 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:23.193254 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:23.290834 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:23.290802 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:23.390122 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:23.390030 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:24.065106 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:24.065072 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:24.164643 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:24.164610 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:24.268825 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:24.268789 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:25.850103 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:25.850071 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:26.360921 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:26.360887 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:32:32.158161 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:32.158131 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:32.233803 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:32.233773 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:32.828181 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:32.828143 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:35.254889 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:35.254852 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:32:38.959802 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:38.959734 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:39.027615 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:39.027580 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:39.209852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:39.209767 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:39.544636 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:39.544588 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:39.914309 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:39.914216 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:42.741620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:42.741584 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:42.821735 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:42.821705 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:42.897142 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:42.897098 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:42.992853 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:42.992753 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:43.285115 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:43.285082 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:43.383207 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:43.383173 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:47.250580 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:47.250551 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:47.912931 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:47.912893 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:48.045900 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:48.045865 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:48.353823 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:48.353792 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:49.560641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:49.560590 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:32:54.120169 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:54.120134 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:54.219450 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:54.219402 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:54.585525 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:54.585497 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:54.993348 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:54.993239 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:55.090348 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:55.090315 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:57.356307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:32:57.356266 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:33:01.171159 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:01.171128 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:01.239907 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:01.239874 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:01.446873 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:01.446790 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:01.825039 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:01.825005 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:01.959885 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:01.959850 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:02.060544 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:02.060506 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:08.226218 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:08.226142 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:08.290275 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:08.290243 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:08.388840 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:08.388799 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:08.479048 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:08.478957 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:08.603247 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:08.603213 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:14.762681 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:14.762641 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:14.862773 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:14.862733 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.570232 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:21.570197 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:21.901084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:21.901003 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:22.008988 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:22.008948 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:24.825091 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:24.825046 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:26.055217 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:26.055179 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:27.453455 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:27.453397 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:33:29.340501 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:29.340469 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:32.079474 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:32.079408 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:32.177566 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:32.177534 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:46.186296 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:46.186263 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:46.286049 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:46.286016 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:50.839046 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:50.839015 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:51.554751 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.554713 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb"] Jul 06 19:33:51.560141 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.560113 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.564440 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.564372 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e2251edc\"/\"kube-root-ca.crt\"" Jul 06 19:33:51.564922 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.564625 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e2251edc\"/\"default-dockercfg-w977x\"" Jul 06 19:33:51.565384 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.565302 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e2251edc\"/\"openshift-service-ca.crt\"" Jul 06 19:33:51.565534 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.565466 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e2251edc\"/\"e2e-lifecycle-model-e2251edc-kserve-self-signed-certs\"" Jul 06 19:33:51.569488 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.569460 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb"] Jul 06 19:33:51.582082 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.582043 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:51.583248 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.583228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-tmp-dir\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.583361 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.583269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-model-cache\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.583442 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.583367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-home\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.583442 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.583400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzm6\" (UniqueName: \"kubernetes.io/projected/c6248890-40c3-4d4a-bb01-74ace31be214-kube-api-access-hlzm6\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.583545 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.583466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6248890-40c3-4d4a-bb01-74ace31be214-tls-certs\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.583545 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.583503 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-dshm\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.684621 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.684580 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-home\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.684621 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.684618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzm6\" (UniqueName: \"kubernetes.io/projected/c6248890-40c3-4d4a-bb01-74ace31be214-kube-api-access-hlzm6\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.684874 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.684641 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6248890-40c3-4d4a-bb01-74ace31be214-tls-certs\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.684874 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.684663 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-dshm\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.684874 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.684698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-tmp-dir\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.684874 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.684738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-model-cache\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.685071 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.685050 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-home\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.685113 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.685069 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-tmp-dir\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.685150 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.685109 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-model-cache\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.686980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.686955 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-dshm\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.687260 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.687240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6248890-40c3-4d4a-bb01-74ace31be214-tls-certs\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.696059 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.696023 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzm6\" (UniqueName: \"kubernetes.io/projected/c6248890-40c3-4d4a-bb01-74ace31be214-kube-api-access-hlzm6\") pod \"e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:51.874003 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:51.873904 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:33:52.014628 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:52.014603 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb"] Jul 06 19:33:52.016457 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:33:52.016405 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6248890_40c3_4d4a_bb01_74ace31be214.slice/crio-4a9e953e26812849fe810616a33e9503f7c417e33d5a899c1ccd835a369e3772 WatchSource:0}: Error finding container 4a9e953e26812849fe810616a33e9503f7c417e33d5a899c1ccd835a369e3772: Status 404 returned error can't find the container with id 4a9e953e26812849fe810616a33e9503f7c417e33d5a899c1ccd835a369e3772 Jul 06 19:33:52.834419 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:52.834381 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" event={"ID":"c6248890-40c3-4d4a-bb01-74ace31be214","Type":"ContainerStarted","Data":"dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505"} Jul 06 19:33:52.834419 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:52.834447 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" event={"ID":"c6248890-40c3-4d4a-bb01-74ace31be214","Type":"ContainerStarted","Data":"4a9e953e26812849fe810616a33e9503f7c417e33d5a899c1ccd835a369e3772"} Jul 06 19:33:52.851827 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:52.851770 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" podStartSLOduration=1.851751022 podStartE2EDuration="1.851751022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:33:52.851088417 +0000 UTC m=+1660.391810645" watchObservedRunningTime="2026-07-06 19:33:52.851751022 +0000 UTC m=+1660.392473252" Jul 06 19:33:54.457145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:33:54.457096 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:34:01.874568 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:01.874527 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:34:01.875040 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:01.874705 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:34:01.887520 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:01.887497 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:34:02.904257 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:02.904226 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:34:19.376410 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:19.376371 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:34:22.144974 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:22.144940 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:22.498638 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:22.498555 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:28.227639 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:28.227599 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:32.360278 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:32.360242 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:34:33.209851 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:33.209813 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb"] Jul 06 19:34:33.303852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:33.303818 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:48.230937 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.230826 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" podUID="c6248890-40c3-4d4a-bb01-74ace31be214" containerName="main" containerID="cri-o://dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505" gracePeriod=45 Jul 06 19:34:48.491110 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.491046 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:34:48.604944 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.604906 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-dshm\") pod \"c6248890-40c3-4d4a-bb01-74ace31be214\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " Jul 06 19:34:48.605145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.604954 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-tmp-dir\") pod \"c6248890-40c3-4d4a-bb01-74ace31be214\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " Jul 06 19:34:48.605145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.604991 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-home\") pod \"c6248890-40c3-4d4a-bb01-74ace31be214\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " Jul 06 19:34:48.605145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.605021 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6248890-40c3-4d4a-bb01-74ace31be214-tls-certs\") pod \"c6248890-40c3-4d4a-bb01-74ace31be214\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " Jul 06 19:34:48.605145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.605052 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlzm6\" (UniqueName: \"kubernetes.io/projected/c6248890-40c3-4d4a-bb01-74ace31be214-kube-api-access-hlzm6\") pod \"c6248890-40c3-4d4a-bb01-74ace31be214\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " Jul 06 19:34:48.605145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.605090 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-model-cache\") pod \"c6248890-40c3-4d4a-bb01-74ace31be214\" (UID: \"c6248890-40c3-4d4a-bb01-74ace31be214\") " Jul 06 19:34:48.605489 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.605293 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-home" (OuterVolumeSpecName: "home") pod "c6248890-40c3-4d4a-bb01-74ace31be214" (UID: "c6248890-40c3-4d4a-bb01-74ace31be214"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:48.605489 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.605310 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6248890-40c3-4d4a-bb01-74ace31be214" (UID: "c6248890-40c3-4d4a-bb01-74ace31be214"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:48.605600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.605484 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:34:48.605600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.605498 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6248890-40c3-4d4a-bb01-74ace31be214" (UID: "c6248890-40c3-4d4a-bb01-74ace31be214"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:48.605600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.605506 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:34:48.607345 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.607308 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-dshm" (OuterVolumeSpecName: "dshm") pod "c6248890-40c3-4d4a-bb01-74ace31be214" (UID: "c6248890-40c3-4d4a-bb01-74ace31be214"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:34:48.607478 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.607377 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6248890-40c3-4d4a-bb01-74ace31be214-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6248890-40c3-4d4a-bb01-74ace31be214" (UID: "c6248890-40c3-4d4a-bb01-74ace31be214"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:34:48.607478 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.607392 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6248890-40c3-4d4a-bb01-74ace31be214-kube-api-access-hlzm6" (OuterVolumeSpecName: "kube-api-access-hlzm6") pod "c6248890-40c3-4d4a-bb01-74ace31be214" (UID: "c6248890-40c3-4d4a-bb01-74ace31be214"). InnerVolumeSpecName "kube-api-access-hlzm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:34:48.706390 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.706351 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6248890-40c3-4d4a-bb01-74ace31be214-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:34:48.706390 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.706383 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlzm6\" (UniqueName: \"kubernetes.io/projected/c6248890-40c3-4d4a-bb01-74ace31be214-kube-api-access-hlzm6\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:34:48.706390 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.706393 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:34:48.706390 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:48.706402 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6248890-40c3-4d4a-bb01-74ace31be214-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:34:49.143772 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.143736 2569 generic.go:356] "Generic (PLEG): container finished" podID="c6248890-40c3-4d4a-bb01-74ace31be214" containerID="dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505" exitCode=0 Jul 06 19:34:49.143961 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.143782 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" Jul 06 19:34:49.143961 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.143830 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" event={"ID":"c6248890-40c3-4d4a-bb01-74ace31be214","Type":"ContainerDied","Data":"dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505"} Jul 06 19:34:49.143961 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.143876 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb" event={"ID":"c6248890-40c3-4d4a-bb01-74ace31be214","Type":"ContainerDied","Data":"4a9e953e26812849fe810616a33e9503f7c417e33d5a899c1ccd835a369e3772"} Jul 06 19:34:49.143961 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.143898 2569 scope.go:117] "RemoveContainer" containerID="dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505" Jul 06 19:34:49.152848 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.152830 2569 scope.go:117] "RemoveContainer" containerID="dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505" Jul 06 19:34:49.153119 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:34:49.153098 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505\": container with ID starting with dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505 not found: ID does not exist" containerID="dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505" Jul 06 19:34:49.153192 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.153129 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505"} err="failed to get container status \"dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505\": rpc error: code = NotFound desc = could not find container \"dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505\": container with ID starting with dc01afa1fe63ebf2b1e3910defed2b57724a9f4b565f3bb64bf76d382acbe505 not found: ID does not exist" Jul 06 19:34:49.160911 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.160874 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb"] Jul 06 19:34:49.164329 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:49.164308 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e2251edc/e2e-lifecycle-model-e2251edc-kserve-7bc4866f9f-spzpb"] Jul 06 19:34:51.022093 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:34:51.022059 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6248890-40c3-4d4a-bb01-74ace31be214" path="/var/lib/kubelet/pods/c6248890-40c3-4d4a-bb01-74ace31be214/volumes" Jul 06 19:35:09.985837 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:09.985802 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:10.741160 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:10.741115 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:13.450626 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:13.450546 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:35:16.447989 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:16.447957 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:16.548135 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:16.548101 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:16.730231 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:16.730144 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:18.003165 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:18.003121 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:18.103020 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:18.102983 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:18.206843 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:18.206812 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:18.986784 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:18.986757 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:19.081398 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:19.081364 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:19.181195 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:19.181165 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:20.512944 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:20.512912 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:22.561557 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:22.561524 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:35:26.692620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:26.692580 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:26.756261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:26.756211 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:27.374760 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:27.374708 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:33.351550 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:33.351517 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:33.352910 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:33.352889 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:35:33.451356 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:33.451320 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:33.556335 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:33.556303 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:44.059259 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:44.059218 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:44.393960 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:35:44.393874 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:05.335987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:05.335905 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:11.461413 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:11.461372 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:11.528917 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:11.528883 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:11.817649 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:11.817618 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:12.166779 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:12.166698 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:12.365534 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:12.365491 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:13.031223 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:13.031189 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:36:13.039413 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:13.039392 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:36:14.729185 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:14.729148 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:14.829693 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:14.829657 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:14.926711 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:14.926680 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:15.333654 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:15.333617 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:15.429209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:15.429171 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:15.528944 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:15.528913 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:19.068230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:19.068197 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:36:19.222372 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:19.222335 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:19.562313 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:19.562281 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:19.999129 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:19.999021 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:20.351203 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:20.351166 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:22.754105 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.754067 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:22.840286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.840250 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8"] Jul 06 19:36:22.840801 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.840780 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6248890-40c3-4d4a-bb01-74ace31be214" containerName="main" Jul 06 19:36:22.840801 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.840800 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6248890-40c3-4d4a-bb01-74ace31be214" containerName="main" Jul 06 19:36:22.840964 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.840887 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6248890-40c3-4d4a-bb01-74ace31be214" containerName="main" Jul 06 19:36:22.841114 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.841090 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:22.844025 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.844006 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:22.845272 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.845241 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a236e9-e705-4b84-9aec-7ec459f71b34-openshift-service-ca\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:22.845392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.845292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:22.845392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.845354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4c6\" (UniqueName: \"kubernetes.io/projected/33a236e9-e705-4b84-9aec-7ec459f71b34-kube-api-access-5q4c6\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:22.847143 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.847122 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 06 19:36:22.847241 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.847144 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 06 19:36:22.847241 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.847131 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gpd9w\"" Jul 06 19:36:22.847558 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.847543 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-67fa37-serving-cert\"" Jul 06 19:36:22.857123 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.857102 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8"] Jul 06 19:36:22.912593 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.912558 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:22.946715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.946683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a236e9-e705-4b84-9aec-7ec459f71b34-openshift-service-ca\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:22.946715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.946719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:22.946959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.946752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4c6\" (UniqueName: \"kubernetes.io/projected/33a236e9-e705-4b84-9aec-7ec459f71b34-kube-api-access-5q4c6\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:22.946959 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:36:22.946886 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-serving-cert: secret "maas-api-e2e-inf-b-67fa37-serving-cert" not found Jul 06 19:36:22.946959 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:36:22.946961 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls podName:33a236e9-e705-4b84-9aec-7ec459f71b34 nodeName:}" failed. No retries permitted until 2026-07-06 19:36:23.446943702 +0000 UTC m=+1810.987665908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls") pod "maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" (UID: "33a236e9-e705-4b84-9aec-7ec459f71b34") : secret "maas-api-e2e-inf-b-67fa37-serving-cert" not found Jul 06 19:36:22.947375 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.947353 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a236e9-e705-4b84-9aec-7ec459f71b34-openshift-service-ca\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:22.959168 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:22.959145 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4c6\" (UniqueName: \"kubernetes.io/projected/33a236e9-e705-4b84-9aec-7ec459f71b34-kube-api-access-5q4c6\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:23.449181 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:23.449147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:23.449364 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:36:23.449265 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-serving-cert: secret "maas-api-e2e-inf-b-67fa37-serving-cert" not found Jul 06 19:36:23.449364 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:36:23.449318 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls podName:33a236e9-e705-4b84-9aec-7ec459f71b34 nodeName:}" failed. No retries permitted until 2026-07-06 19:36:24.44930306 +0000 UTC m=+1811.990025267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls") pod "maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" (UID: "33a236e9-e705-4b84-9aec-7ec459f71b34") : secret "maas-api-e2e-inf-b-67fa37-serving-cert" not found Jul 06 19:36:23.522218 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:23.522187 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:23.622790 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:23.622757 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:24.455498 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:24.455457 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:24.457955 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:24.457924 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls\") pod \"maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:24.654542 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:24.654502 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:24.791578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:24.791534 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8"] Jul 06 19:36:24.793366 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:36:24.793326 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a236e9_e705_4b84_9aec_7ec459f71b34.slice/crio-43896bb361d5f5abd177b580b2e2bf62ef804eb73e11d574f8f9aa41e1189a70 WatchSource:0}: Error finding container 43896bb361d5f5abd177b580b2e2bf62ef804eb73e11d574f8f9aa41e1189a70: Status 404 returned error can't find the container with id 43896bb361d5f5abd177b580b2e2bf62ef804eb73e11d574f8f9aa41e1189a70 Jul 06 19:36:24.795358 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:24.795333 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:36:25.677308 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:25.677263 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" event={"ID":"33a236e9-e705-4b84-9aec-7ec459f71b34","Type":"ContainerStarted","Data":"43896bb361d5f5abd177b580b2e2bf62ef804eb73e11d574f8f9aa41e1189a70"} Jul 06 19:36:26.684628 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:26.684542 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" event={"ID":"33a236e9-e705-4b84-9aec-7ec459f71b34","Type":"ContainerStarted","Data":"aad8aab50cae186f774503c645d6488cbe7f179eccd6d5f3e222fc4a8212fc77"} Jul 06 19:36:26.684992 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:26.684669 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:26.703203 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:26.703153 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" podStartSLOduration=3.071611462 podStartE2EDuration="4.703136779s" totalImagesPullingTime="1.631525317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:26.701311836 +0000 UTC m=+1814.242034065" watchObservedRunningTime="2026-07-06 19:36:26.703136779 +0000 UTC m=+1814.243859006" Jul 06 19:36:27.247756 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:27.247726 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:29.755356 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:29.755320 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:36:32.695227 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:32.695195 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:36:35.652931 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.652897 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j"] Jul 06 19:36:35.656795 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.656771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.660414 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.660386 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-90ac12\"/\"openshift-service-ca.crt\"" Jul 06 19:36:35.660585 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.660418 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-90ac12\"/\"kube-root-ca.crt\"" Jul 06 19:36:35.660585 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.660440 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-90ac12\"/\"test-model-90ac12-kserve-self-signed-certs\"" Jul 06 19:36:35.660585 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.660418 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-90ac12\"/\"default-dockercfg-mv827\"" Jul 06 19:36:35.667555 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.667530 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j"] Jul 06 19:36:35.676202 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.676166 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:35.750751 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.750714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-tmp-dir\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.750751 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.750750 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30eeb257-1810-42cc-831e-e8fb6f3968d0-tls-certs\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.750968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.750845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-home\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.750968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.750880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbtk7\" (UniqueName: \"kubernetes.io/projected/30eeb257-1810-42cc-831e-e8fb6f3968d0-kube-api-access-hbtk7\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.750968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.750906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-model-cache\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.750968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.750943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-dshm\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852350 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtk7\" (UniqueName: \"kubernetes.io/projected/30eeb257-1810-42cc-831e-e8fb6f3968d0-kube-api-access-hbtk7\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852350 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-model-cache\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852636 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852419 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-dshm\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852636 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-tmp-dir\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852636 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852539 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30eeb257-1810-42cc-831e-e8fb6f3968d0-tls-certs\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852636 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852608 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-home\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852846 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852800 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-model-cache\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852919 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852896 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-tmp-dir\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.852962 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.852921 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-home\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.854819 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.854800 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-dshm\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.855128 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.855111 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30eeb257-1810-42cc-831e-e8fb6f3968d0-tls-certs\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.860738 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.860718 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtk7\" (UniqueName: \"kubernetes.io/projected/30eeb257-1810-42cc-831e-e8fb6f3968d0-kube-api-access-hbtk7\") pod \"test-model-90ac12-kserve-dd7b86fbf-tt28j\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:35.966872 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:35.966759 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:36.108578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:36.108550 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j"] Jul 06 19:36:36.111158 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:36:36.111131 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30eeb257_1810_42cc_831e_e8fb6f3968d0.slice/crio-6d86d60d9ecb81b006ad9da6afab016362a3c8dd9ed65bc1237c995a44a2f9ae WatchSource:0}: Error finding container 6d86d60d9ecb81b006ad9da6afab016362a3c8dd9ed65bc1237c995a44a2f9ae: Status 404 returned error can't find the container with id 6d86d60d9ecb81b006ad9da6afab016362a3c8dd9ed65bc1237c995a44a2f9ae Jul 06 19:36:36.207707 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:36.207668 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:36.307997 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:36.307962 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:36.741934 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:36.741828 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" event={"ID":"30eeb257-1810-42cc-831e-e8fb6f3968d0","Type":"ContainerStarted","Data":"19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8"} Jul 06 19:36:36.741934 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:36.741884 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" event={"ID":"30eeb257-1810-42cc-831e-e8fb6f3968d0","Type":"ContainerStarted","Data":"6d86d60d9ecb81b006ad9da6afab016362a3c8dd9ed65bc1237c995a44a2f9ae"} Jul 06 19:36:36.757981 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:36.757918 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" podStartSLOduration=1.757896693 podStartE2EDuration="1.757896693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:36.757217124 +0000 UTC m=+1824.297939352" watchObservedRunningTime="2026-07-06 19:36:36.757896693 +0000 UTC m=+1824.298618923" Jul 06 19:36:39.062467 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:39.062417 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:36:45.473621 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:45.473587 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:45.556561 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:45.556521 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:45.967251 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:45.967215 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:45.967474 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:45.967309 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:45.979664 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:45.979642 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:46.809398 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:46.809366 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:36:49.855755 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:36:49.855719 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:37:00.038842 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.038805 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57"] Jul 06 19:37:00.042686 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.042666 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.045011 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.044988 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-67fa37\"/\"kube-root-ca.crt\"" Jul 06 19:37:00.045851 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.045826 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-67fa37\"/\"default-dockercfg-2sx9c\"" Jul 06 19:37:00.045851 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.045844 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-67fa37\"/\"openshift-service-ca.crt\"" Jul 06 19:37:00.046007 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.045869 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-67fa37\"/\"test-model-67fa37-kserve-self-signed-certs\"" Jul 06 19:37:00.054129 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.054107 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57"] Jul 06 19:37:00.064041 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.064018 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:00.174676 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.174635 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-dshm\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.174865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.174683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-tmp-dir\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.174865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.174748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-model-cache\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.174865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.174794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd2e39f-cb99-4a23-b80e-53c36107d091-tls-certs\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.174865 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.174831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-home\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.175057 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.174879 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rklnq\" (UniqueName: \"kubernetes.io/projected/ddd2e39f-cb99-4a23-b80e-53c36107d091-kube-api-access-rklnq\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276085 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-dshm\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276085 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-tmp-dir\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-model-cache\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd2e39f-cb99-4a23-b80e-53c36107d091-tls-certs\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-home\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rklnq\" (UniqueName: \"kubernetes.io/projected/ddd2e39f-cb99-4a23-b80e-53c36107d091-kube-api-access-rklnq\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276631 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-home\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276631 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276625 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-model-cache\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.276740 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.276719 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-tmp-dir\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.278522 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.278504 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-dshm\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.278793 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.278776 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd2e39f-cb99-4a23-b80e-53c36107d091-tls-certs\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.285022 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.284999 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rklnq\" (UniqueName: \"kubernetes.io/projected/ddd2e39f-cb99-4a23-b80e-53c36107d091-kube-api-access-rklnq\") pod \"test-model-67fa37-kserve-6bc5449965-zmk57\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.353917 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.353817 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:00.499873 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.499839 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57"] Jul 06 19:37:00.500642 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:37:00.500613 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddd2e39f_cb99_4a23_b80e_53c36107d091.slice/crio-2caca7f014c48ac1816968f8702e8f02819a6bc08ccc5452038c9372aade5d04 WatchSource:0}: Error finding container 2caca7f014c48ac1816968f8702e8f02819a6bc08ccc5452038c9372aade5d04: Status 404 returned error can't find the container with id 2caca7f014c48ac1816968f8702e8f02819a6bc08ccc5452038c9372aade5d04 Jul 06 19:37:00.876481 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.876419 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" event={"ID":"ddd2e39f-cb99-4a23-b80e-53c36107d091","Type":"ContainerStarted","Data":"f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66"} Jul 06 19:37:00.876481 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.876486 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" event={"ID":"ddd2e39f-cb99-4a23-b80e-53c36107d091","Type":"ContainerStarted","Data":"2caca7f014c48ac1816968f8702e8f02819a6bc08ccc5452038c9372aade5d04"} Jul 06 19:37:00.897639 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:00.897590 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" podStartSLOduration=0.897574733 podStartE2EDuration="897.574733ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:00.895652555 +0000 UTC m=+1848.436374784" watchObservedRunningTime="2026-07-06 19:37:00.897574733 +0000 UTC m=+1848.438296961" Jul 06 19:37:04.154947 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:04.154911 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:37:10.354060 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:10.354018 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:10.354500 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:10.354080 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:10.367009 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:10.366975 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:10.945764 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:10.945735 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:16.764538 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:16.764495 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:37:20.542445 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:20.542385 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j"] Jul 06 19:37:20.563889 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:20.563855 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:21.536601 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:21.536566 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57"] Jul 06 19:37:21.558674 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:21.558642 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:21.702110 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:21.702074 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:22.083174 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:22.083144 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:22.186408 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:22.186380 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:22.284559 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:22.284525 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:22.394830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:22.394751 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:27.166742 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:27.166686 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:31.852927 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:31.852893 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:37:35.562798 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.562696 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" podUID="30eeb257-1810-42cc-831e-e8fb6f3968d0" containerName="main" containerID="cri-o://19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8" gracePeriod=45 Jul 06 19:37:35.820656 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.820594 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:37:35.888034 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888000 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbtk7\" (UniqueName: \"kubernetes.io/projected/30eeb257-1810-42cc-831e-e8fb6f3968d0-kube-api-access-hbtk7\") pod \"30eeb257-1810-42cc-831e-e8fb6f3968d0\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " Jul 06 19:37:35.888228 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888067 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30eeb257-1810-42cc-831e-e8fb6f3968d0-tls-certs\") pod \"30eeb257-1810-42cc-831e-e8fb6f3968d0\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " Jul 06 19:37:35.888228 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888097 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-home\") pod \"30eeb257-1810-42cc-831e-e8fb6f3968d0\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " Jul 06 19:37:35.888228 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888125 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-tmp-dir\") pod \"30eeb257-1810-42cc-831e-e8fb6f3968d0\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " Jul 06 19:37:35.888228 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888145 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-dshm\") pod \"30eeb257-1810-42cc-831e-e8fb6f3968d0\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " Jul 06 19:37:35.888228 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888187 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-model-cache\") pod \"30eeb257-1810-42cc-831e-e8fb6f3968d0\" (UID: \"30eeb257-1810-42cc-831e-e8fb6f3968d0\") " Jul 06 19:37:35.888534 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888391 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-home" (OuterVolumeSpecName: "home") pod "30eeb257-1810-42cc-831e-e8fb6f3968d0" (UID: "30eeb257-1810-42cc-831e-e8fb6f3968d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:37:35.888534 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888413 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30eeb257-1810-42cc-831e-e8fb6f3968d0" (UID: "30eeb257-1810-42cc-831e-e8fb6f3968d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:37:35.888534 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888487 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:35.888638 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.888557 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "30eeb257-1810-42cc-831e-e8fb6f3968d0" (UID: "30eeb257-1810-42cc-831e-e8fb6f3968d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:37:35.890416 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.890393 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30eeb257-1810-42cc-831e-e8fb6f3968d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30eeb257-1810-42cc-831e-e8fb6f3968d0" (UID: "30eeb257-1810-42cc-831e-e8fb6f3968d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:37:35.890501 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.890446 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-dshm" (OuterVolumeSpecName: "dshm") pod "30eeb257-1810-42cc-831e-e8fb6f3968d0" (UID: "30eeb257-1810-42cc-831e-e8fb6f3968d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:37:35.890539 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.890500 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30eeb257-1810-42cc-831e-e8fb6f3968d0-kube-api-access-hbtk7" (OuterVolumeSpecName: "kube-api-access-hbtk7") pod "30eeb257-1810-42cc-831e-e8fb6f3968d0" (UID: "30eeb257-1810-42cc-831e-e8fb6f3968d0"). InnerVolumeSpecName "kube-api-access-hbtk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:37:35.989411 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.989373 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:35.989411 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.989405 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:35.989411 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.989414 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30eeb257-1810-42cc-831e-e8fb6f3968d0-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:35.989411 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.989445 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbtk7\" (UniqueName: \"kubernetes.io/projected/30eeb257-1810-42cc-831e-e8fb6f3968d0-kube-api-access-hbtk7\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:35.989752 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:35.989455 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30eeb257-1810-42cc-831e-e8fb6f3968d0-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:36.073339 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.073256 2569 generic.go:356] "Generic (PLEG): container finished" podID="30eeb257-1810-42cc-831e-e8fb6f3968d0" containerID="19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8" exitCode=0 Jul 06 19:37:36.073339 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.073299 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" Jul 06 19:37:36.073615 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.073338 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" event={"ID":"30eeb257-1810-42cc-831e-e8fb6f3968d0","Type":"ContainerDied","Data":"19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8"} Jul 06 19:37:36.073615 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.073380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j" event={"ID":"30eeb257-1810-42cc-831e-e8fb6f3968d0","Type":"ContainerDied","Data":"6d86d60d9ecb81b006ad9da6afab016362a3c8dd9ed65bc1237c995a44a2f9ae"} Jul 06 19:37:36.073615 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.073397 2569 scope.go:117] "RemoveContainer" containerID="19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8" Jul 06 19:37:36.082821 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.082803 2569 scope.go:117] "RemoveContainer" containerID="19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8" Jul 06 19:37:36.083066 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:37:36.083050 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8\": container with ID starting with 19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8 not found: ID does not exist" containerID="19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8" Jul 06 19:37:36.083106 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.083074 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8"} err="failed to get container status \"19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8\": rpc error: code = NotFound desc = could not find container \"19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8\": container with ID starting with 19a961bb0d955fbfaa488ad9bc2d40ce4cbdff0260bae689892a4c5bc3f787e8 not found: ID does not exist" Jul 06 19:37:36.095657 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.095628 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j"] Jul 06 19:37:36.101309 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.101285 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-90ac12/test-model-90ac12-kserve-dd7b86fbf-tt28j"] Jul 06 19:37:36.557571 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.557532 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" podUID="ddd2e39f-cb99-4a23-b80e-53c36107d091" containerName="main" containerID="cri-o://f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66" gracePeriod=45 Jul 06 19:37:36.815805 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.815739 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:36.898281 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898250 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-home\") pod \"ddd2e39f-cb99-4a23-b80e-53c36107d091\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " Jul 06 19:37:36.898498 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898304 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-tmp-dir\") pod \"ddd2e39f-cb99-4a23-b80e-53c36107d091\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " Jul 06 19:37:36.898498 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898330 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd2e39f-cb99-4a23-b80e-53c36107d091-tls-certs\") pod \"ddd2e39f-cb99-4a23-b80e-53c36107d091\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " Jul 06 19:37:36.898498 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898349 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rklnq\" (UniqueName: \"kubernetes.io/projected/ddd2e39f-cb99-4a23-b80e-53c36107d091-kube-api-access-rklnq\") pod \"ddd2e39f-cb99-4a23-b80e-53c36107d091\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " Jul 06 19:37:36.898498 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898403 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-dshm\") pod \"ddd2e39f-cb99-4a23-b80e-53c36107d091\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " Jul 06 19:37:36.898498 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898419 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-model-cache\") pod \"ddd2e39f-cb99-4a23-b80e-53c36107d091\" (UID: \"ddd2e39f-cb99-4a23-b80e-53c36107d091\") " Jul 06 19:37:36.898780 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898587 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-home" (OuterVolumeSpecName: "home") pod "ddd2e39f-cb99-4a23-b80e-53c36107d091" (UID: "ddd2e39f-cb99-4a23-b80e-53c36107d091"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:37:36.898780 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898727 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:36.898780 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898748 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddd2e39f-cb99-4a23-b80e-53c36107d091" (UID: "ddd2e39f-cb99-4a23-b80e-53c36107d091"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:37:36.898984 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.898956 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddd2e39f-cb99-4a23-b80e-53c36107d091" (UID: "ddd2e39f-cb99-4a23-b80e-53c36107d091"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:37:36.900594 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.900573 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd2e39f-cb99-4a23-b80e-53c36107d091-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddd2e39f-cb99-4a23-b80e-53c36107d091" (UID: "ddd2e39f-cb99-4a23-b80e-53c36107d091"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:37:36.900929 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.900910 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-dshm" (OuterVolumeSpecName: "dshm") pod "ddd2e39f-cb99-4a23-b80e-53c36107d091" (UID: "ddd2e39f-cb99-4a23-b80e-53c36107d091"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:37:36.900997 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.900916 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd2e39f-cb99-4a23-b80e-53c36107d091-kube-api-access-rklnq" (OuterVolumeSpecName: "kube-api-access-rklnq") pod "ddd2e39f-cb99-4a23-b80e-53c36107d091" (UID: "ddd2e39f-cb99-4a23-b80e-53c36107d091"). InnerVolumeSpecName "kube-api-access-rklnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:37:36.999602 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.999565 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:36.999602 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.999603 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:36.999819 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.999617 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd2e39f-cb99-4a23-b80e-53c36107d091-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:36.999819 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.999630 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd2e39f-cb99-4a23-b80e-53c36107d091-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:36.999819 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:36.999642 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rklnq\" (UniqueName: \"kubernetes.io/projected/ddd2e39f-cb99-4a23-b80e-53c36107d091-kube-api-access-rklnq\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:37:37.022261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.022226 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30eeb257-1810-42cc-831e-e8fb6f3968d0" path="/var/lib/kubelet/pods/30eeb257-1810-42cc-831e-e8fb6f3968d0/volumes" Jul 06 19:37:37.081468 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.081353 2569 generic.go:356] "Generic (PLEG): container finished" podID="ddd2e39f-cb99-4a23-b80e-53c36107d091" containerID="f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66" exitCode=0 Jul 06 19:37:37.081468 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.081399 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" Jul 06 19:37:37.081710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.081453 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" event={"ID":"ddd2e39f-cb99-4a23-b80e-53c36107d091","Type":"ContainerDied","Data":"f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66"} Jul 06 19:37:37.081710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.081521 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57" event={"ID":"ddd2e39f-cb99-4a23-b80e-53c36107d091","Type":"ContainerDied","Data":"2caca7f014c48ac1816968f8702e8f02819a6bc08ccc5452038c9372aade5d04"} Jul 06 19:37:37.081710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.081543 2569 scope.go:117] "RemoveContainer" containerID="f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66" Jul 06 19:37:37.090812 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.090792 2569 scope.go:117] "RemoveContainer" containerID="f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66" Jul 06 19:37:37.091101 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:37:37.091074 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66\": container with ID starting with f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66 not found: ID does not exist" containerID="f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66" Jul 06 19:37:37.091149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.091104 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66"} err="failed to get container status \"f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66\": rpc error: code = NotFound desc = could not find container \"f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66\": container with ID starting with f2b2179ff028c7c974f703f8e25badfd886da80a10c5e02f758a120d40ccbd66 not found: ID does not exist" Jul 06 19:37:37.100907 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.100885 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57"] Jul 06 19:37:37.103214 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:37.103190 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-67fa37/test-model-67fa37-kserve-6bc5449965-zmk57"] Jul 06 19:37:39.029363 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:37:39.029326 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd2e39f-cb99-4a23-b80e-53c36107d091" path="/var/lib/kubelet/pods/ddd2e39f-cb99-4a23-b80e-53c36107d091/volumes" Jul 06 19:38:04.496790 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:04.496753 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:04.762973 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:04.762939 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:05.277245 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:05.277216 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:05.627076 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:05.626994 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:05.727209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:05.727171 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:05.827968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:05.827935 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:05.908956 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:05.908859 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8"] Jul 06 19:38:05.909528 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:05.909479 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" podUID="33a236e9-e705-4b84-9aec-7ec459f71b34" containerName="maas-api" containerID="cri-o://aad8aab50cae186f774503c645d6488cbe7f179eccd6d5f3e222fc4a8212fc77" gracePeriod=30 Jul 06 19:38:05.943709 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:05.943678 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:06.247099 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.247070 2569 generic.go:356] "Generic (PLEG): container finished" podID="33a236e9-e705-4b84-9aec-7ec459f71b34" containerID="aad8aab50cae186f774503c645d6488cbe7f179eccd6d5f3e222fc4a8212fc77" exitCode=0 Jul 06 19:38:06.247258 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.247153 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" event={"ID":"33a236e9-e705-4b84-9aec-7ec459f71b34","Type":"ContainerDied","Data":"aad8aab50cae186f774503c645d6488cbe7f179eccd6d5f3e222fc4a8212fc77"} Jul 06 19:38:06.359047 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.359022 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:38:06.469185 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.469090 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a236e9-e705-4b84-9aec-7ec459f71b34-openshift-service-ca\") pod \"33a236e9-e705-4b84-9aec-7ec459f71b34\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " Jul 06 19:38:06.469185 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.469156 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls\") pod \"33a236e9-e705-4b84-9aec-7ec459f71b34\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " Jul 06 19:38:06.469406 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.469200 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q4c6\" (UniqueName: \"kubernetes.io/projected/33a236e9-e705-4b84-9aec-7ec459f71b34-kube-api-access-5q4c6\") pod \"33a236e9-e705-4b84-9aec-7ec459f71b34\" (UID: \"33a236e9-e705-4b84-9aec-7ec459f71b34\") " Jul 06 19:38:06.469519 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.469473 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a236e9-e705-4b84-9aec-7ec459f71b34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33a236e9-e705-4b84-9aec-7ec459f71b34" (UID: "33a236e9-e705-4b84-9aec-7ec459f71b34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:38:06.471413 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.471383 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33a236e9-e705-4b84-9aec-7ec459f71b34" (UID: "33a236e9-e705-4b84-9aec-7ec459f71b34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:38:06.471552 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.471461 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a236e9-e705-4b84-9aec-7ec459f71b34-kube-api-access-5q4c6" (OuterVolumeSpecName: "kube-api-access-5q4c6") pod "33a236e9-e705-4b84-9aec-7ec459f71b34" (UID: "33a236e9-e705-4b84-9aec-7ec459f71b34"). InnerVolumeSpecName "kube-api-access-5q4c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:38:06.570351 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.570295 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a236e9-e705-4b84-9aec-7ec459f71b34-openshift-service-ca\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:38:06.570351 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.570344 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a236e9-e705-4b84-9aec-7ec459f71b34-maas-api-tls\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:38:06.570351 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:06.570354 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q4c6\" (UniqueName: \"kubernetes.io/projected/33a236e9-e705-4b84-9aec-7ec459f71b34-kube-api-access-5q4c6\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:38:07.253564 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:07.253528 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" event={"ID":"33a236e9-e705-4b84-9aec-7ec459f71b34","Type":"ContainerDied","Data":"43896bb361d5f5abd177b580b2e2bf62ef804eb73e11d574f8f9aa41e1189a70"} Jul 06 19:38:07.254024 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:07.253584 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8" Jul 06 19:38:07.254024 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:07.253612 2569 scope.go:117] "RemoveContainer" containerID="aad8aab50cae186f774503c645d6488cbe7f179eccd6d5f3e222fc4a8212fc77" Jul 06 19:38:07.271335 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:07.271309 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8"] Jul 06 19:38:07.275101 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:07.275077 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-67fa37-6dc7955d9c-2zkr8"] Jul 06 19:38:09.022060 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:09.022029 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a236e9-e705-4b84-9aec-7ec459f71b34" path="/var/lib/kubelet/pods/33a236e9-e705-4b84-9aec-7ec459f71b34/volumes" Jul 06 19:38:09.150679 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:09.150644 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:38:11.832667 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:11.832629 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:11.933386 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:11.933350 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:12.123804 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:12.123719 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:12.665959 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:12.665918 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:13.026531 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:13.026498 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:13.588092 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:13.588057 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:15.900289 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.900258 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:15.965130 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965095 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j"] Jul 06 19:38:15.965774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965741 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a236e9-e705-4b84-9aec-7ec459f71b34" containerName="maas-api" Jul 06 19:38:15.965774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965762 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a236e9-e705-4b84-9aec-7ec459f71b34" containerName="maas-api" Jul 06 19:38:15.965774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965775 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30eeb257-1810-42cc-831e-e8fb6f3968d0" containerName="main" Jul 06 19:38:15.965987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965784 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30eeb257-1810-42cc-831e-e8fb6f3968d0" containerName="main" Jul 06 19:38:15.965987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965796 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd2e39f-cb99-4a23-b80e-53c36107d091" containerName="main" Jul 06 19:38:15.965987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965805 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd2e39f-cb99-4a23-b80e-53c36107d091" containerName="main" Jul 06 19:38:15.965987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965903 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30eeb257-1810-42cc-831e-e8fb6f3968d0" containerName="main" Jul 06 19:38:15.965987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965918 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd2e39f-cb99-4a23-b80e-53c36107d091" containerName="main" Jul 06 19:38:15.965987 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.965928 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a236e9-e705-4b84-9aec-7ec459f71b34" containerName="maas-api" Jul 06 19:38:15.973273 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.973251 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:15.975679 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.975652 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8f1080-serving-cert\"" Jul 06 19:38:15.975816 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.975710 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gpd9w\"" Jul 06 19:38:15.975816 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.975738 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 06 19:38:15.976480 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.976461 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 06 19:38:15.982035 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.982009 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j"] Jul 06 19:38:15.985637 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:15.985614 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:16.056185 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.056143 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:16.067466 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.067414 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695212b0-7f27-4acc-aa9e-93429000dbd0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.067660 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.067494 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.067660 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.067581 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5n5\" (UniqueName: \"kubernetes.io/projected/695212b0-7f27-4acc-aa9e-93429000dbd0-kube-api-access-2w5n5\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.168315 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.168213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695212b0-7f27-4acc-aa9e-93429000dbd0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.168315 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.168269 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.168578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.168324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5n5\" (UniqueName: \"kubernetes.io/projected/695212b0-7f27-4acc-aa9e-93429000dbd0-kube-api-access-2w5n5\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.168578 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:38:16.168465 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-serving-cert: secret "maas-api-e2e-api-a-8f1080-serving-cert" not found Jul 06 19:38:16.168578 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:38:16.168563 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls podName:695212b0-7f27-4acc-aa9e-93429000dbd0 nodeName:}" failed. No retries permitted until 2026-07-06 19:38:16.668538129 +0000 UTC m=+1924.209260351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls") pod "maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" (UID: "695212b0-7f27-4acc-aa9e-93429000dbd0") : secret "maas-api-e2e-api-a-8f1080-serving-cert" not found Jul 06 19:38:16.168948 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.168929 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695212b0-7f27-4acc-aa9e-93429000dbd0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.180228 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.180196 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5n5\" (UniqueName: \"kubernetes.io/projected/695212b0-7f27-4acc-aa9e-93429000dbd0-kube-api-access-2w5n5\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.520357 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.520326 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:16.657673 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.657637 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:16.671906 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.671877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.674326 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.674297 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls\") pod \"maas-api-e2e-api-a-8f1080-cb89c7798-dl49j\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:16.884684 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:16.884591 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:17.043175 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:17.043148 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j"] Jul 06 19:38:17.044922 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:38:17.044896 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695212b0_7f27_4acc_aa9e_93429000dbd0.slice/crio-c83d22e2351fb252ca89406eb792b9fa0c577293d7d80c8b7fa4273e7e73b617 WatchSource:0}: Error finding container c83d22e2351fb252ca89406eb792b9fa0c577293d7d80c8b7fa4273e7e73b617: Status 404 returned error can't find the container with id c83d22e2351fb252ca89406eb792b9fa0c577293d7d80c8b7fa4273e7e73b617 Jul 06 19:38:17.312981 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:17.312941 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" event={"ID":"695212b0-7f27-4acc-aa9e-93429000dbd0","Type":"ContainerStarted","Data":"c83d22e2351fb252ca89406eb792b9fa0c577293d7d80c8b7fa4273e7e73b617"} Jul 06 19:38:18.320500 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:18.320464 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" event={"ID":"695212b0-7f27-4acc-aa9e-93429000dbd0","Type":"ContainerStarted","Data":"82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e"} Jul 06 19:38:18.320888 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:18.320562 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:18.341246 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:18.341193 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" podStartSLOduration=2.92923225 podStartE2EDuration="3.341177833s" totalImagesPullingTime="411.945583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:18.339878335 +0000 UTC m=+1925.880600574" watchObservedRunningTime="2026-07-06 19:38:18.341177833 +0000 UTC m=+1925.881900084" Jul 06 19:38:20.421541 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:20.421476 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:20.858858 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:20.858823 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:21.164265 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:21.164176 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:21.309610 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:21.309572 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:22.756640 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:22.756608 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:38:24.331667 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:24.331639 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:24.375279 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:24.375218 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:24.478627 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:24.478587 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:24.626793 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:24.626692 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:24.884186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:24.884099 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:25.228012 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:25.227913 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:25.326622 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:25.326582 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:26.740700 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:26.740661 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:28.351593 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:28.351564 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:28.860137 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:28.860102 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:32.859827 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:32.859794 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:32.949713 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:32.949677 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:33.289812 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:33.289784 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:33.391716 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:33.391677 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:34.252066 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:34.252031 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:38:36.860592 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:36.860554 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:37.315742 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:37.315705 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:37.498861 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:37.498828 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:37.598463 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:37.598364 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:37.744548 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:37.744517 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j"] Jul 06 19:38:37.744801 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:37.744772 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" podUID="695212b0-7f27-4acc-aa9e-93429000dbd0" containerName="maas-api" containerID="cri-o://82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e" gracePeriod=30 Jul 06 19:38:37.765576 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:37.765549 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:37.999532 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:37.999503 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:38.027636 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.027606 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w5n5\" (UniqueName: \"kubernetes.io/projected/695212b0-7f27-4acc-aa9e-93429000dbd0-kube-api-access-2w5n5\") pod \"695212b0-7f27-4acc-aa9e-93429000dbd0\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " Jul 06 19:38:38.027810 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.027670 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695212b0-7f27-4acc-aa9e-93429000dbd0-openshift-service-ca\") pod \"695212b0-7f27-4acc-aa9e-93429000dbd0\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " Jul 06 19:38:38.027810 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.027768 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls\") pod \"695212b0-7f27-4acc-aa9e-93429000dbd0\" (UID: \"695212b0-7f27-4acc-aa9e-93429000dbd0\") " Jul 06 19:38:38.028100 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.028076 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695212b0-7f27-4acc-aa9e-93429000dbd0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "695212b0-7f27-4acc-aa9e-93429000dbd0" (UID: "695212b0-7f27-4acc-aa9e-93429000dbd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:38:38.029930 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.029903 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695212b0-7f27-4acc-aa9e-93429000dbd0-kube-api-access-2w5n5" (OuterVolumeSpecName: "kube-api-access-2w5n5") pod "695212b0-7f27-4acc-aa9e-93429000dbd0" (UID: "695212b0-7f27-4acc-aa9e-93429000dbd0"). InnerVolumeSpecName "kube-api-access-2w5n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:38:38.030223 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.030197 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "695212b0-7f27-4acc-aa9e-93429000dbd0" (UID: "695212b0-7f27-4acc-aa9e-93429000dbd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:38:38.128620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.128534 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695212b0-7f27-4acc-aa9e-93429000dbd0-openshift-service-ca\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:38:38.128620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.128567 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695212b0-7f27-4acc-aa9e-93429000dbd0-maas-api-tls\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:38:38.128620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.128578 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w5n5\" (UniqueName: \"kubernetes.io/projected/695212b0-7f27-4acc-aa9e-93429000dbd0-kube-api-access-2w5n5\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:38:38.448160 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.448066 2569 generic.go:356] "Generic (PLEG): container finished" podID="695212b0-7f27-4acc-aa9e-93429000dbd0" containerID="82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e" exitCode=0 Jul 06 19:38:38.448160 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.448116 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" Jul 06 19:38:38.448409 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.448156 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" event={"ID":"695212b0-7f27-4acc-aa9e-93429000dbd0","Type":"ContainerDied","Data":"82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e"} Jul 06 19:38:38.448409 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.448218 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j" event={"ID":"695212b0-7f27-4acc-aa9e-93429000dbd0","Type":"ContainerDied","Data":"c83d22e2351fb252ca89406eb792b9fa0c577293d7d80c8b7fa4273e7e73b617"} Jul 06 19:38:38.448409 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.448247 2569 scope.go:117] "RemoveContainer" containerID="82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e" Jul 06 19:38:38.461742 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.461721 2569 scope.go:117] "RemoveContainer" containerID="82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e" Jul 06 19:38:38.462073 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:38:38.462053 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e\": container with ID starting with 82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e not found: ID does not exist" containerID="82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e" Jul 06 19:38:38.462152 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.462083 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e"} err="failed to get container status \"82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e\": rpc error: code = NotFound desc = could not find container \"82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e\": container with ID starting with 82d21bd59809560bfd15e7beda0f98e64813116df3ccd5b7fa386e28f1121e0e not found: ID does not exist" Jul 06 19:38:38.472563 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.472534 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j"] Jul 06 19:38:38.478396 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:38.478369 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8f1080-cb89c7798-dl49j"] Jul 06 19:38:39.021631 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:39.021600 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695212b0-7f27-4acc-aa9e-93429000dbd0" path="/var/lib/kubelet/pods/695212b0-7f27-4acc-aa9e-93429000dbd0/volumes" Jul 06 19:38:43.513384 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:43.513351 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:43.585979 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:43.585947 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:43.771825 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:43.771746 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:44.246407 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:44.246333 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:44.346248 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:44.346195 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:44.549293 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:44.549259 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:44.649850 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:44.649822 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:44.750516 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:44.750482 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:44.857932 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:44.857850 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:47.959351 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:47.959315 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:38:50.360123 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:50.360084 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:50.424310 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:50.424275 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:50.620064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:50.619965 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:51.157662 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:51.157629 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:51.456783 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:51.456697 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:51.559150 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:51.559114 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:54.539831 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.539798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:54.604337 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.604304 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6"] Jul 06 19:38:54.604733 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.604717 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695212b0-7f27-4acc-aa9e-93429000dbd0" containerName="maas-api" Jul 06 19:38:54.604782 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.604735 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="695212b0-7f27-4acc-aa9e-93429000dbd0" containerName="maas-api" Jul 06 19:38:54.604818 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.604799 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="695212b0-7f27-4acc-aa9e-93429000dbd0" containerName="maas-api" Jul 06 19:38:54.608232 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.608207 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.611131 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.611108 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 06 19:38:54.611131 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.611115 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-f969f4-serving-cert\"" Jul 06 19:38:54.611342 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.611302 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 06 19:38:54.611935 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.611918 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gpd9w\"" Jul 06 19:38:54.618545 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.618512 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6"] Jul 06 19:38:54.620857 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.620823 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:54.685833 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.685802 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:54.689962 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.689936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653eb53c-28ad-4a87-8ad4-04c23037652c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.690095 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.689973 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653eb53c-28ad-4a87-8ad4-04c23037652c-maas-api-tls\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.690095 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.690001 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmhp\" (UniqueName: \"kubernetes.io/projected/653eb53c-28ad-4a87-8ad4-04c23037652c-kube-api-access-rrmhp\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.787819 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.787791 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:54.791014 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.790962 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653eb53c-28ad-4a87-8ad4-04c23037652c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.791014 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.790997 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653eb53c-28ad-4a87-8ad4-04c23037652c-maas-api-tls\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.791014 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.791023 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmhp\" (UniqueName: \"kubernetes.io/projected/653eb53c-28ad-4a87-8ad4-04c23037652c-kube-api-access-rrmhp\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.791683 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.791659 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653eb53c-28ad-4a87-8ad4-04c23037652c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.793497 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.793478 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653eb53c-28ad-4a87-8ad4-04c23037652c-maas-api-tls\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.799244 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.799210 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmhp\" (UniqueName: \"kubernetes.io/projected/653eb53c-28ad-4a87-8ad4-04c23037652c-kube-api-access-rrmhp\") pod \"maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:54.920408 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:54.920357 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:55.069792 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:55.069766 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6"] Jul 06 19:38:55.071794 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:38:55.071768 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653eb53c_28ad_4a87_8ad4_04c23037652c.slice/crio-21c891e8fae72633ddc223334920305857f0bc9ba2918f9515b0f93b324547a8 WatchSource:0}: Error finding container 21c891e8fae72633ddc223334920305857f0bc9ba2918f9515b0f93b324547a8: Status 404 returned error can't find the container with id 21c891e8fae72633ddc223334920305857f0bc9ba2918f9515b0f93b324547a8 Jul 06 19:38:55.209845 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:55.209814 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:55.310592 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:55.310559 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:55.545943 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:55.545914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" event={"ID":"653eb53c-28ad-4a87-8ad4-04c23037652c","Type":"ContainerStarted","Data":"21c891e8fae72633ddc223334920305857f0bc9ba2918f9515b0f93b324547a8"} Jul 06 19:38:56.553028 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:56.552990 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" event={"ID":"653eb53c-28ad-4a87-8ad4-04c23037652c","Type":"ContainerStarted","Data":"45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c"} Jul 06 19:38:56.553461 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:56.553107 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:38:56.570974 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:56.570928 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" podStartSLOduration=2.149359304 podStartE2EDuration="2.570911629s" totalImagesPullingTime="421.552325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:56.568719037 +0000 UTC m=+1964.109441266" watchObservedRunningTime="2026-07-06 19:38:56.570911629 +0000 UTC m=+1964.111633860" Jul 06 19:38:59.050279 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:59.050243 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:59.296180 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:59.296137 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:38:59.381486 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:59.381360 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:59.674944 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:38:59.674868 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:00.040327 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:00.040297 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:02.564185 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:02.564159 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:39:02.604741 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:02.604707 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:02.674554 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:02.674519 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:02.929734 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:02.929653 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:03.056802 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:03.056770 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:03.414651 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:03.414621 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:03.515394 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:03.515346 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:07.075776 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:07.075694 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:07.848922 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:07.848888 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:39:16.097687 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.097653 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:16.197976 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.197940 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:16.858066 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.858037 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq"] Jul 06 19:39:16.861500 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.861463 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.864835 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.864814 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"openshift-service-ca.crt\"" Jul 06 19:39:16.864968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.864815 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"default-dockercfg-zplsd\"" Jul 06 19:39:16.864968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.864815 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"auth-test-model-f969f4-kserve-self-signed-certs\"" Jul 06 19:39:16.864968 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.864820 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"kube-root-ca.crt\"" Jul 06 19:39:16.869781 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.869750 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq"] Jul 06 19:39:16.885155 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.885123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tls-certs\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.885291 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.885184 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-dshm\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.885291 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.885205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-model-cache\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.885291 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.885221 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-home\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.885291 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.885239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tmp-dir\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.885442 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.885292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxrq\" (UniqueName: \"kubernetes.io/projected/d603e0b5-088e-4664-9c46-4c8f7a275e6f-kube-api-access-2hxrq\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.899271 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.899242 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:16.986704 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.986662 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-dshm\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.986704 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.986704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-model-cache\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.986975 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.986744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-home\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.986975 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.986772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tmp-dir\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.986975 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.986800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxrq\" (UniqueName: \"kubernetes.io/projected/d603e0b5-088e-4664-9c46-4c8f7a275e6f-kube-api-access-2hxrq\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.986975 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.986850 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tls-certs\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.987179 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.987122 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-model-cache\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.987259 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.987215 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-home\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.987307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.987256 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tmp-dir\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.989149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.989120 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-dshm\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.989327 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.989309 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tls-certs\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:16.995878 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:16.995851 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxrq\" (UniqueName: \"kubernetes.io/projected/d603e0b5-088e-4664-9c46-4c8f7a275e6f-kube-api-access-2hxrq\") pod \"auth-test-model-f969f4-kserve-557fc5d947-qqqzq\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:17.172492 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:17.172405 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:17.314742 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:17.314702 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq"] Jul 06 19:39:17.316528 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:39:17.316494 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd603e0b5_088e_4664_9c46_4c8f7a275e6f.slice/crio-d4950965b17664c4183d90ecd0097323da9a6e107352cc52c93bf777932f953b WatchSource:0}: Error finding container d4950965b17664c4183d90ecd0097323da9a6e107352cc52c93bf777932f953b: Status 404 returned error can't find the container with id d4950965b17664c4183d90ecd0097323da9a6e107352cc52c93bf777932f953b Jul 06 19:39:17.670068 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:17.670033 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" event={"ID":"d603e0b5-088e-4664-9c46-4c8f7a275e6f","Type":"ContainerStarted","Data":"d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1"} Jul 06 19:39:17.670068 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:17.670072 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" event={"ID":"d603e0b5-088e-4664-9c46-4c8f7a275e6f","Type":"ContainerStarted","Data":"d4950965b17664c4183d90ecd0097323da9a6e107352cc52c93bf777932f953b"} Jul 06 19:39:17.691581 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:17.691526 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" podStartSLOduration=1.69150944 podStartE2EDuration="1.69150944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:17.689314211 +0000 UTC m=+1985.230036439" watchObservedRunningTime="2026-07-06 19:39:17.69150944 +0000 UTC m=+1985.232231668" Jul 06 19:39:20.658513 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:20.658475 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:39:24.124592 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:24.124559 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:24.227042 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:24.227008 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:27.173177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:27.173149 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:27.173633 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:27.173196 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:27.185245 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:27.185220 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:27.734809 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:27.734771 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:39:54.379506 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:54.379474 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:56.861745 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.861707 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp"] Jul 06 19:39:56.865539 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.865505 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:56.868906 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.868880 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"kube-root-ca.crt\"" Jul 06 19:39:56.868906 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.868895 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"auth-test-model-f31ab4-kserve-self-signed-certs\"" Jul 06 19:39:56.869116 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.868932 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"openshift-service-ca.crt\"" Jul 06 19:39:56.869116 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.868997 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"default-dockercfg-5ln7f\"" Jul 06 19:39:56.875415 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.875394 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp"] Jul 06 19:39:56.944553 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.944518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597z2\" (UniqueName: \"kubernetes.io/projected/16979da2-3a28-46ae-8c15-b343c2546fe4-kube-api-access-597z2\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:56.944719 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.944558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-dshm\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:56.944719 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.944587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-home\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:56.944719 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.944669 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-tmp-dir\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:56.944719 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.944707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-model-cache\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:56.944860 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.944797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16979da2-3a28-46ae-8c15-b343c2546fe4-tls-certs\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:56.951257 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:56.951225 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:57.045578 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.045541 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16979da2-3a28-46ae-8c15-b343c2546fe4-tls-certs\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.045750 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.045592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597z2\" (UniqueName: \"kubernetes.io/projected/16979da2-3a28-46ae-8c15-b343c2546fe4-kube-api-access-597z2\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.045750 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.045706 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-dshm\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.045750 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.045742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-home\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.045888 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.045869 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-tmp-dir\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.045932 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.045916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-model-cache\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.046122 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.046102 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-home\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.046201 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.046163 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-tmp-dir\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.046201 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.046196 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-model-cache\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.047832 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.047814 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-dshm\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.048229 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.048206 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16979da2-3a28-46ae-8c15-b343c2546fe4-tls-certs\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.053967 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.053944 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-597z2\" (UniqueName: \"kubernetes.io/projected/16979da2-3a28-46ae-8c15-b343c2546fe4-kube-api-access-597z2\") pod \"auth-test-model-f31ab4-kserve-6d857b5fc9-chztp\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.176633 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.176547 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:39:57.319396 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.319368 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp"] Jul 06 19:39:57.320924 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:39:57.320901 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16979da2_3a28_46ae_8c15_b343c2546fe4.slice/crio-ef02fe52d74b28db55596a0d04d8f6518b9b0b43a64d410cb5364f732b0eff7a WatchSource:0}: Error finding container ef02fe52d74b28db55596a0d04d8f6518b9b0b43a64d410cb5364f732b0eff7a: Status 404 returned error can't find the container with id ef02fe52d74b28db55596a0d04d8f6518b9b0b43a64d410cb5364f732b0eff7a Jul 06 19:39:57.885382 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.885347 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" event={"ID":"16979da2-3a28-46ae-8c15-b343c2546fe4","Type":"ContainerStarted","Data":"1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454"} Jul 06 19:39:57.885382 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.885388 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" event={"ID":"16979da2-3a28-46ae-8c15-b343c2546fe4","Type":"ContainerStarted","Data":"ef02fe52d74b28db55596a0d04d8f6518b9b0b43a64d410cb5364f732b0eff7a"} Jul 06 19:39:57.901289 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:39:57.901234 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" podStartSLOduration=1.9012163279999998 podStartE2EDuration="1.901216328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:39:57.900583768 +0000 UTC m=+2025.441306035" watchObservedRunningTime="2026-07-06 19:39:57.901216328 +0000 UTC m=+2025.441938625" Jul 06 19:40:00.363843 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:00.363807 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:40:07.177153 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:07.177118 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:40:07.177660 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:07.177167 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:40:07.192551 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:07.192526 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:40:07.954385 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:07.954357 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:40:12.399246 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:12.399199 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:40:26.750137 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:26.750100 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:40:43.354188 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:43.354114 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:40:53.416189 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:53.416155 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:58.456871 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:40:58.456834 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:41:13.073649 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:13.073619 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:41:13.083241 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:13.083218 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:41:13.651156 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:13.651119 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:41:29.468376 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:29.468342 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:41:46.754718 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:46.754689 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:41:50.530656 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:50.530612 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq"] Jul 06 19:41:50.550225 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:50.550197 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:51.471342 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:51.471314 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp"] Jul 06 19:41:51.493641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:41:51.493492 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:01.848358 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:01.848316 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:42:05.550347 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.550235 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" podUID="d603e0b5-088e-4664-9c46-4c8f7a275e6f" containerName="main" containerID="cri-o://d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1" gracePeriod=45 Jul 06 19:42:05.792663 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.792637 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:42:05.912402 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912325 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-dshm\") pod \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " Jul 06 19:42:05.912402 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912369 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-model-cache\") pod \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " Jul 06 19:42:05.912640 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912472 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tls-certs\") pod \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " Jul 06 19:42:05.912640 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912511 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hxrq\" (UniqueName: \"kubernetes.io/projected/d603e0b5-088e-4664-9c46-4c8f7a275e6f-kube-api-access-2hxrq\") pod \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " Jul 06 19:42:05.912640 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912542 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tmp-dir\") pod \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " Jul 06 19:42:05.912640 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912574 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-home\") pod \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\" (UID: \"d603e0b5-088e-4664-9c46-4c8f7a275e6f\") " Jul 06 19:42:05.912640 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912607 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d603e0b5-088e-4664-9c46-4c8f7a275e6f" (UID: "d603e0b5-088e-4664-9c46-4c8f7a275e6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:42:05.912900 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912864 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:05.912900 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912869 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-home" (OuterVolumeSpecName: "home") pod "d603e0b5-088e-4664-9c46-4c8f7a275e6f" (UID: "d603e0b5-088e-4664-9c46-4c8f7a275e6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:42:05.912982 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.912907 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d603e0b5-088e-4664-9c46-4c8f7a275e6f" (UID: "d603e0b5-088e-4664-9c46-4c8f7a275e6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:42:05.914662 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.914639 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d603e0b5-088e-4664-9c46-4c8f7a275e6f-kube-api-access-2hxrq" (OuterVolumeSpecName: "kube-api-access-2hxrq") pod "d603e0b5-088e-4664-9c46-4c8f7a275e6f" (UID: "d603e0b5-088e-4664-9c46-4c8f7a275e6f"). InnerVolumeSpecName "kube-api-access-2hxrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:42:05.914773 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.914756 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d603e0b5-088e-4664-9c46-4c8f7a275e6f" (UID: "d603e0b5-088e-4664-9c46-4c8f7a275e6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:42:05.914826 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:05.914774 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-dshm" (OuterVolumeSpecName: "dshm") pod "d603e0b5-088e-4664-9c46-4c8f7a275e6f" (UID: "d603e0b5-088e-4664-9c46-4c8f7a275e6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:42:06.013296 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.013264 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.013296 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.013291 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxrq\" (UniqueName: \"kubernetes.io/projected/d603e0b5-088e-4664-9c46-4c8f7a275e6f-kube-api-access-2hxrq\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.013296 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.013303 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.013566 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.013311 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.013566 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.013318 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d603e0b5-088e-4664-9c46-4c8f7a275e6f-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.491478 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.491393 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" podUID="16979da2-3a28-46ae-8c15-b343c2546fe4" containerName="main" containerID="cri-o://1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454" gracePeriod=45 Jul 06 19:42:06.589805 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.589773 2569 generic.go:356] "Generic (PLEG): container finished" podID="d603e0b5-088e-4664-9c46-4c8f7a275e6f" containerID="d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1" exitCode=0 Jul 06 19:42:06.590222 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.589814 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" Jul 06 19:42:06.590222 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.589853 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" event={"ID":"d603e0b5-088e-4664-9c46-4c8f7a275e6f","Type":"ContainerDied","Data":"d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1"} Jul 06 19:42:06.590222 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.589891 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq" event={"ID":"d603e0b5-088e-4664-9c46-4c8f7a275e6f","Type":"ContainerDied","Data":"d4950965b17664c4183d90ecd0097323da9a6e107352cc52c93bf777932f953b"} Jul 06 19:42:06.590222 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.589907 2569 scope.go:117] "RemoveContainer" containerID="d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1" Jul 06 19:42:06.603647 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.603626 2569 scope.go:117] "RemoveContainer" containerID="d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1" Jul 06 19:42:06.603919 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:42:06.603901 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1\": container with ID starting with d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1 not found: ID does not exist" containerID="d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1" Jul 06 19:42:06.603992 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.603935 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1"} err="failed to get container status \"d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1\": rpc error: code = NotFound desc = could not find container \"d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1\": container with ID starting with d259a93b2f0e66fbb5867c109ed6c67b8c38fa57ad1f394f28c28a5711992ad1 not found: ID does not exist" Jul 06 19:42:06.614262 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.614234 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq"] Jul 06 19:42:06.616878 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.616854 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/auth-test-model-f969f4-kserve-557fc5d947-qqqzq"] Jul 06 19:42:06.754858 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.754800 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:42:06.820695 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.820664 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-home\") pod \"16979da2-3a28-46ae-8c15-b343c2546fe4\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " Jul 06 19:42:06.820882 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.820704 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-model-cache\") pod \"16979da2-3a28-46ae-8c15-b343c2546fe4\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " Jul 06 19:42:06.820882 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.820742 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-dshm\") pod \"16979da2-3a28-46ae-8c15-b343c2546fe4\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " Jul 06 19:42:06.820882 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.820796 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-tmp-dir\") pod \"16979da2-3a28-46ae-8c15-b343c2546fe4\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " Jul 06 19:42:06.820882 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.820821 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16979da2-3a28-46ae-8c15-b343c2546fe4-tls-certs\") pod \"16979da2-3a28-46ae-8c15-b343c2546fe4\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " Jul 06 19:42:06.820882 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.820884 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-597z2\" (UniqueName: \"kubernetes.io/projected/16979da2-3a28-46ae-8c15-b343c2546fe4-kube-api-access-597z2\") pod \"16979da2-3a28-46ae-8c15-b343c2546fe4\" (UID: \"16979da2-3a28-46ae-8c15-b343c2546fe4\") " Jul 06 19:42:06.821151 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.820969 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-home" (OuterVolumeSpecName: "home") pod "16979da2-3a28-46ae-8c15-b343c2546fe4" (UID: "16979da2-3a28-46ae-8c15-b343c2546fe4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:42:06.821151 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.821009 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16979da2-3a28-46ae-8c15-b343c2546fe4" (UID: "16979da2-3a28-46ae-8c15-b343c2546fe4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:42:06.821151 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.821039 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-model-cache" (OuterVolumeSpecName: "model-cache") pod "16979da2-3a28-46ae-8c15-b343c2546fe4" (UID: "16979da2-3a28-46ae-8c15-b343c2546fe4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:42:06.821278 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.821152 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.821278 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.821169 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.821278 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.821183 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.822976 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.822955 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-dshm" (OuterVolumeSpecName: "dshm") pod "16979da2-3a28-46ae-8c15-b343c2546fe4" (UID: "16979da2-3a28-46ae-8c15-b343c2546fe4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:42:06.823049 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.823010 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16979da2-3a28-46ae-8c15-b343c2546fe4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16979da2-3a28-46ae-8c15-b343c2546fe4" (UID: "16979da2-3a28-46ae-8c15-b343c2546fe4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:42:06.823085 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.823047 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16979da2-3a28-46ae-8c15-b343c2546fe4-kube-api-access-597z2" (OuterVolumeSpecName: "kube-api-access-597z2") pod "16979da2-3a28-46ae-8c15-b343c2546fe4" (UID: "16979da2-3a28-46ae-8c15-b343c2546fe4"). InnerVolumeSpecName "kube-api-access-597z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:42:06.922001 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.921966 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-597z2\" (UniqueName: \"kubernetes.io/projected/16979da2-3a28-46ae-8c15-b343c2546fe4-kube-api-access-597z2\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.922001 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.921995 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16979da2-3a28-46ae-8c15-b343c2546fe4-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:06.922001 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:06.922005 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16979da2-3a28-46ae-8c15-b343c2546fe4-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:42:07.028003 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.027972 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d603e0b5-088e-4664-9c46-4c8f7a275e6f" path="/var/lib/kubelet/pods/d603e0b5-088e-4664-9c46-4c8f7a275e6f/volumes" Jul 06 19:42:07.598205 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.598166 2569 generic.go:356] "Generic (PLEG): container finished" podID="16979da2-3a28-46ae-8c15-b343c2546fe4" containerID="1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454" exitCode=0 Jul 06 19:42:07.598726 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.598235 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" Jul 06 19:42:07.598726 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.598248 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" event={"ID":"16979da2-3a28-46ae-8c15-b343c2546fe4","Type":"ContainerDied","Data":"1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454"} Jul 06 19:42:07.598726 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.598320 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp" event={"ID":"16979da2-3a28-46ae-8c15-b343c2546fe4","Type":"ContainerDied","Data":"ef02fe52d74b28db55596a0d04d8f6518b9b0b43a64d410cb5364f732b0eff7a"} Jul 06 19:42:07.598726 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.598343 2569 scope.go:117] "RemoveContainer" containerID="1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454" Jul 06 19:42:07.608093 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.608074 2569 scope.go:117] "RemoveContainer" containerID="1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454" Jul 06 19:42:07.608371 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:42:07.608352 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454\": container with ID starting with 1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454 not found: ID does not exist" containerID="1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454" Jul 06 19:42:07.608447 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.608380 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454"} err="failed to get container status \"1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454\": rpc error: code = NotFound desc = could not find container \"1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454\": container with ID starting with 1f68ebecd06c2a627a487b5aa35ff9e904724a7b085160e271e5325f3f3b7454 not found: ID does not exist" Jul 06 19:42:07.615572 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.615544 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp"] Jul 06 19:42:07.619086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:07.619065 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/auth-test-model-f31ab4-kserve-6d857b5fc9-chztp"] Jul 06 19:42:08.342585 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.342555 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8"] Jul 06 19:42:08.342944 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.342932 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16979da2-3a28-46ae-8c15-b343c2546fe4" containerName="main" Jul 06 19:42:08.342944 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.342945 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="16979da2-3a28-46ae-8c15-b343c2546fe4" containerName="main" Jul 06 19:42:08.343027 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.342964 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d603e0b5-088e-4664-9c46-4c8f7a275e6f" containerName="main" Jul 06 19:42:08.343027 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.342970 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d603e0b5-088e-4664-9c46-4c8f7a275e6f" containerName="main" Jul 06 19:42:08.343088 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.343033 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="16979da2-3a28-46ae-8c15-b343c2546fe4" containerName="main" Jul 06 19:42:08.343088 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.343042 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d603e0b5-088e-4664-9c46-4c8f7a275e6f" containerName="main" Jul 06 19:42:08.347648 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.347631 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.350924 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.350900 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"openshift-service-ca.crt\"" Jul 06 19:42:08.351067 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.350970 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"default-dockercfg-zplsd\"" Jul 06 19:42:08.351067 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.350976 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"sub-test-model-f969f4-kserve-self-signed-certs\"" Jul 06 19:42:08.351067 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.350991 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"kube-root-ca.crt\"" Jul 06 19:42:08.357674 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.357655 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8"] Jul 06 19:42:08.358665 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.358641 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:08.436582 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.436550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tls-certs\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.436794 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.436601 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-model-cache\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.436794 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.436720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tmp-dir\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.436794 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.436764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl45j\" (UniqueName: \"kubernetes.io/projected/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-kube-api-access-fl45j\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.437010 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.436824 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-dshm\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.437010 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.436873 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-home\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.537919 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.537888 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-home\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.537919 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.537924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tls-certs\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.538119 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.537973 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-model-cache\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.538156 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.538123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tmp-dir\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.538187 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.538152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl45j\" (UniqueName: \"kubernetes.io/projected/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-kube-api-access-fl45j\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.538231 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.538215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-dshm\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.538356 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.538338 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-home\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.538356 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.538349 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-model-cache\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.538467 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.538446 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tmp-dir\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.540208 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.540188 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-dshm\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.540501 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.540483 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tls-certs\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.548261 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.548240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl45j\" (UniqueName: \"kubernetes.io/projected/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-kube-api-access-fl45j\") pod \"sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.658163 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.658080 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:08.790925 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:08.790896 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8"] Jul 06 19:42:08.794113 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:42:08.794083 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66823efc_fe3e_4dc2_a3cb_8272f5c4a8e5.slice/crio-2955b9caf5c35ef02f7a27fb7e6a1f0acdc98cb72340bd4173358bf444ab759d WatchSource:0}: Error finding container 2955b9caf5c35ef02f7a27fb7e6a1f0acdc98cb72340bd4173358bf444ab759d: Status 404 returned error can't find the container with id 2955b9caf5c35ef02f7a27fb7e6a1f0acdc98cb72340bd4173358bf444ab759d Jul 06 19:42:09.021746 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:09.021711 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16979da2-3a28-46ae-8c15-b343c2546fe4" path="/var/lib/kubelet/pods/16979da2-3a28-46ae-8c15-b343c2546fe4/volumes" Jul 06 19:42:09.613793 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:09.613757 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" event={"ID":"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5","Type":"ContainerStarted","Data":"7b0ee77081bb0cb27386ff0cec445571ef67fcfdcd56bd18dc0c28657d52472b"} Jul 06 19:42:09.613963 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:09.613799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" event={"ID":"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5","Type":"ContainerStarted","Data":"2955b9caf5c35ef02f7a27fb7e6a1f0acdc98cb72340bd4173358bf444ab759d"} Jul 06 19:42:09.632842 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:09.632791 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" podStartSLOduration=1.632777077 podStartE2EDuration="1.632777077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:09.630491908 +0000 UTC m=+2157.171214137" watchObservedRunningTime="2026-07-06 19:42:09.632777077 +0000 UTC m=+2157.173499304" Jul 06 19:42:12.761550 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:12.761507 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:42:18.659173 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:18.659119 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:18.659173 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:18.659186 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:18.671586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:18.671559 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:18.682643 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:18.682617 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:42:45.738140 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.738108 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg"] Jul 06 19:42:45.741766 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.741744 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.744161 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.744142 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"openshift-service-ca.crt\"" Jul 06 19:42:45.744414 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.744398 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"default-dockercfg-5ln7f\"" Jul 06 19:42:45.745356 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.745338 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"kube-root-ca.crt\"" Jul 06 19:42:45.745464 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.745338 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"sub-test-model-f31ab4-kserve-self-signed-certs\"" Jul 06 19:42:45.751021 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.751001 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg"] Jul 06 19:42:45.773084 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.773056 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:42:45.866308 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.866278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-model-cache\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.866308 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.866311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-tmp-dir\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.866563 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.866332 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb9655c-2735-4506-b079-df93319373d6-tls-certs\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.866563 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.866355 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-dshm\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.866563 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.866478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-home\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.866563 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.866527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcv5d\" (UniqueName: \"kubernetes.io/projected/2fb9655c-2735-4506-b079-df93319373d6-kube-api-access-tcv5d\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967449 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967406 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-home\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967596 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcv5d\" (UniqueName: \"kubernetes.io/projected/2fb9655c-2735-4506-b079-df93319373d6-kube-api-access-tcv5d\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967596 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-model-cache\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967596 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967549 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-tmp-dir\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967596 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb9655c-2735-4506-b079-df93319373d6-tls-certs\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967815 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-dshm\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967877 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967836 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-home\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967933 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967889 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-model-cache\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.967981 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.967962 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-tmp-dir\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.969711 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.969692 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-dshm\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.970024 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.970006 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb9655c-2735-4506-b079-df93319373d6-tls-certs\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:45.975600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:45.975584 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcv5d\" (UniqueName: \"kubernetes.io/projected/2fb9655c-2735-4506-b079-df93319373d6-kube-api-access-tcv5d\") pod \"sub-test-model-f31ab4-kserve-595477f9df-sqrsg\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:46.052381 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:46.052355 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:46.182655 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:46.182630 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg"] Jul 06 19:42:46.184638 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:42:46.184608 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fb9655c_2735_4506_b079_df93319373d6.slice/crio-306d1a0b85ab8dd20cf605ff370038951fd7a44d22f107c0378113517bf324ad WatchSource:0}: Error finding container 306d1a0b85ab8dd20cf605ff370038951fd7a44d22f107c0378113517bf324ad: Status 404 returned error can't find the container with id 306d1a0b85ab8dd20cf605ff370038951fd7a44d22f107c0378113517bf324ad Jul 06 19:42:46.814441 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:46.814397 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" event={"ID":"2fb9655c-2735-4506-b079-df93319373d6","Type":"ContainerStarted","Data":"55a7f8c0c5bc3dc03edde8a585719998addb120aaba8b5f77edbd6074f312885"} Jul 06 19:42:46.814914 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:46.814450 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" event={"ID":"2fb9655c-2735-4506-b079-df93319373d6","Type":"ContainerStarted","Data":"306d1a0b85ab8dd20cf605ff370038951fd7a44d22f107c0378113517bf324ad"} Jul 06 19:42:46.831081 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:46.831036 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" podStartSLOduration=1.831021945 podStartE2EDuration="1.831021945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:42:46.829718625 +0000 UTC m=+2194.370440854" watchObservedRunningTime="2026-07-06 19:42:46.831021945 +0000 UTC m=+2194.371744172" Jul 06 19:42:50.152010 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:50.151974 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:42:56.053508 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:56.053464 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:56.053954 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:56.053520 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:56.065808 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:56.065788 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:42:56.885531 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:42:56.885502 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:43:14.954477 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:14.954441 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:43:31.849743 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:31.849711 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:43:33.621359 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:33.621316 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:35.326025 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:35.325950 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:37.046147 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:37.046110 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8"] Jul 06 19:43:37.059888 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:37.059856 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:37.960238 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:37.960208 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg"] Jul 06 19:43:37.978830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:37.978785 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:47.453951 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:47.453912 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:43:52.066459 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.066385 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" podUID="66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" containerName="main" containerID="cri-o://7b0ee77081bb0cb27386ff0cec445571ef67fcfdcd56bd18dc0c28657d52472b" gracePeriod=45 Jul 06 19:43:52.186643 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.186614 2569 generic.go:356] "Generic (PLEG): container finished" podID="66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" containerID="7b0ee77081bb0cb27386ff0cec445571ef67fcfdcd56bd18dc0c28657d52472b" exitCode=0 Jul 06 19:43:52.186772 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.186687 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" event={"ID":"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5","Type":"ContainerDied","Data":"7b0ee77081bb0cb27386ff0cec445571ef67fcfdcd56bd18dc0c28657d52472b"} Jul 06 19:43:52.331989 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.331930 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:43:52.446324 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446288 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-dshm\") pod \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " Jul 06 19:43:52.446519 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446333 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl45j\" (UniqueName: \"kubernetes.io/projected/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-kube-api-access-fl45j\") pod \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " Jul 06 19:43:52.446519 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446361 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tls-certs\") pod \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " Jul 06 19:43:52.446519 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446412 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-model-cache\") pod \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " Jul 06 19:43:52.446519 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446465 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tmp-dir\") pod \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " Jul 06 19:43:52.446519 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446514 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-home\") pod \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\" (UID: \"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5\") " Jul 06 19:43:52.446774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446694 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" (UID: "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:52.446774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446739 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" (UID: "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:52.446892 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446816 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-home" (OuterVolumeSpecName: "home") pod "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" (UID: "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:52.446892 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446885 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:52.446980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.446901 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:52.448586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.448556 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" (UID: "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:43:52.448867 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.448845 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-dshm" (OuterVolumeSpecName: "dshm") pod "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" (UID: "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:52.448867 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.448856 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-kube-api-access-fl45j" (OuterVolumeSpecName: "kube-api-access-fl45j") pod "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" (UID: "66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5"). InnerVolumeSpecName "kube-api-access-fl45j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:52.547496 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.547459 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:52.547496 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.547487 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:52.547710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.547503 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl45j\" (UniqueName: \"kubernetes.io/projected/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-kube-api-access-fl45j\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:52.547710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.547517 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:52.980490 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:52.980415 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" podUID="2fb9655c-2735-4506-b079-df93319373d6" containerName="main" containerID="cri-o://55a7f8c0c5bc3dc03edde8a585719998addb120aaba8b5f77edbd6074f312885" gracePeriod=45 Jul 06 19:43:53.194886 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.194856 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" Jul 06 19:43:53.195346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.194914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8" event={"ID":"66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5","Type":"ContainerDied","Data":"2955b9caf5c35ef02f7a27fb7e6a1f0acdc98cb72340bd4173358bf444ab759d"} Jul 06 19:43:53.195346 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.194965 2569 scope.go:117] "RemoveContainer" containerID="7b0ee77081bb0cb27386ff0cec445571ef67fcfdcd56bd18dc0c28657d52472b" Jul 06 19:43:53.197832 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.197743 2569 generic.go:356] "Generic (PLEG): container finished" podID="2fb9655c-2735-4506-b079-df93319373d6" containerID="55a7f8c0c5bc3dc03edde8a585719998addb120aaba8b5f77edbd6074f312885" exitCode=0 Jul 06 19:43:53.197832 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.197783 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" event={"ID":"2fb9655c-2735-4506-b079-df93319373d6","Type":"ContainerDied","Data":"55a7f8c0c5bc3dc03edde8a585719998addb120aaba8b5f77edbd6074f312885"} Jul 06 19:43:53.213847 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.213821 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8"] Jul 06 19:43:53.220249 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.220227 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/sub-test-model-f969f4-kserve-7cb5dd67d7-zfgh8"] Jul 06 19:43:53.242319 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.242266 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:43:53.353953 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.353925 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-dshm\") pod \"2fb9655c-2735-4506-b079-df93319373d6\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " Jul 06 19:43:53.353953 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.353954 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcv5d\" (UniqueName: \"kubernetes.io/projected/2fb9655c-2735-4506-b079-df93319373d6-kube-api-access-tcv5d\") pod \"2fb9655c-2735-4506-b079-df93319373d6\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " Jul 06 19:43:53.354189 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.353984 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-model-cache\") pod \"2fb9655c-2735-4506-b079-df93319373d6\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " Jul 06 19:43:53.354189 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.354043 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-home\") pod \"2fb9655c-2735-4506-b079-df93319373d6\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " Jul 06 19:43:53.354189 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.354064 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb9655c-2735-4506-b079-df93319373d6-tls-certs\") pod \"2fb9655c-2735-4506-b079-df93319373d6\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " Jul 06 19:43:53.354189 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.354100 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-tmp-dir\") pod \"2fb9655c-2735-4506-b079-df93319373d6\" (UID: \"2fb9655c-2735-4506-b079-df93319373d6\") " Jul 06 19:43:53.354409 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.354336 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fb9655c-2735-4506-b079-df93319373d6" (UID: "2fb9655c-2735-4506-b079-df93319373d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:53.354409 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.354328 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-home" (OuterVolumeSpecName: "home") pod "2fb9655c-2735-4506-b079-df93319373d6" (UID: "2fb9655c-2735-4506-b079-df93319373d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:53.354528 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.354417 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fb9655c-2735-4506-b079-df93319373d6" (UID: "2fb9655c-2735-4506-b079-df93319373d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:53.356211 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.356189 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-dshm" (OuterVolumeSpecName: "dshm") pod "2fb9655c-2735-4506-b079-df93319373d6" (UID: "2fb9655c-2735-4506-b079-df93319373d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:53.356211 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.356196 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb9655c-2735-4506-b079-df93319373d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fb9655c-2735-4506-b079-df93319373d6" (UID: "2fb9655c-2735-4506-b079-df93319373d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:43:53.356358 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.356230 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb9655c-2735-4506-b079-df93319373d6-kube-api-access-tcv5d" (OuterVolumeSpecName: "kube-api-access-tcv5d") pod "2fb9655c-2735-4506-b079-df93319373d6" (UID: "2fb9655c-2735-4506-b079-df93319373d6"). InnerVolumeSpecName "kube-api-access-tcv5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:53.455418 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.455387 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:53.455418 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.455415 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:53.455641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.455447 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcv5d\" (UniqueName: \"kubernetes.io/projected/2fb9655c-2735-4506-b079-df93319373d6-kube-api-access-tcv5d\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:53.455641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.455458 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:53.455641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.455466 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb9655c-2735-4506-b079-df93319373d6-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:53.455641 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:53.455473 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb9655c-2735-4506-b079-df93319373d6-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:43:54.205830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:54.205769 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" event={"ID":"2fb9655c-2735-4506-b079-df93319373d6","Type":"ContainerDied","Data":"306d1a0b85ab8dd20cf605ff370038951fd7a44d22f107c0378113517bf324ad"} Jul 06 19:43:54.205830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:54.205800 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg" Jul 06 19:43:54.205830 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:54.205815 2569 scope.go:117] "RemoveContainer" containerID="55a7f8c0c5bc3dc03edde8a585719998addb120aaba8b5f77edbd6074f312885" Jul 06 19:43:54.228036 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:54.228011 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg"] Jul 06 19:43:54.231541 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:54.231508 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/sub-test-model-f31ab4-kserve-595477f9df-sqrsg"] Jul 06 19:43:55.027038 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:55.027005 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb9655c-2735-4506-b079-df93319373d6" path="/var/lib/kubelet/pods/2fb9655c-2735-4506-b079-df93319373d6/volumes" Jul 06 19:43:55.027377 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:43:55.027365 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" path="/var/lib/kubelet/pods/66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5/volumes" Jul 06 19:44:08.256582 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.256549 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg"] Jul 06 19:44:08.256956 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.256928 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" containerName="main" Jul 06 19:44:08.256956 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.256938 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" containerName="main" Jul 06 19:44:08.256956 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.256946 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb9655c-2735-4506-b079-df93319373d6" containerName="main" Jul 06 19:44:08.256956 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.256951 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb9655c-2735-4506-b079-df93319373d6" containerName="main" Jul 06 19:44:08.257086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.257017 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66823efc-fe3e-4dc2-a3cb-8272f5c4a8e5" containerName="main" Jul 06 19:44:08.257086 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.257027 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb9655c-2735-4506-b079-df93319373d6" containerName="main" Jul 06 19:44:08.261601 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.261584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.263840 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.263822 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"openshift-service-ca.crt\"" Jul 06 19:44:08.263933 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.263919 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"default-dockercfg-zplsd\"" Jul 06 19:44:08.264710 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.264691 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"kube-root-ca.crt\"" Jul 06 19:44:08.264774 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.264696 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f969f4\"/\"rate-test-model-f969f4-kserve-self-signed-certs\"" Jul 06 19:44:08.269246 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.269225 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg"] Jul 06 19:44:08.273011 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.272990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:08.377384 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.377346 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-dshm\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.377586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.377480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tls-certs\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.377586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.377504 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tmp-dir\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.377586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.377564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-home\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.377735 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.377643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-model-cache\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.377735 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.377665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7nq\" (UniqueName: \"kubernetes.io/projected/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-kube-api-access-vn7nq\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.478879 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.478841 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-dshm\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.479074 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.478942 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tls-certs\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.479074 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.478971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tmp-dir\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.479074 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.478998 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-home\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.479074 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.479033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-model-cache\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.479074 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.479054 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7nq\" (UniqueName: \"kubernetes.io/projected/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-kube-api-access-vn7nq\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.479390 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.479354 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tmp-dir\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.479390 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.479371 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-home\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.479521 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.479408 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-model-cache\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.481173 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.481154 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-dshm\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.481335 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.481317 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tls-certs\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.487537 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.487508 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7nq\" (UniqueName: \"kubernetes.io/projected/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-kube-api-access-vn7nq\") pod \"rate-test-model-f969f4-kserve-757ff5f68c-8mprg\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.571841 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.571812 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:08.704666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:08.704638 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg"] Jul 06 19:44:08.706610 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:44:08.706583 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d2c6d4f_bdd2_48c8_b4e8_6c03523aceb4.slice/crio-a4f88e3e8f141084d73ee0565ed21c2ade30019a9df1dc3c3ac749702e4c9742 WatchSource:0}: Error finding container a4f88e3e8f141084d73ee0565ed21c2ade30019a9df1dc3c3ac749702e4c9742: Status 404 returned error can't find the container with id a4f88e3e8f141084d73ee0565ed21c2ade30019a9df1dc3c3ac749702e4c9742 Jul 06 19:44:09.295941 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:09.295907 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" event={"ID":"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4","Type":"ContainerStarted","Data":"538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6"} Jul 06 19:44:09.295941 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:09.295945 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" event={"ID":"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4","Type":"ContainerStarted","Data":"a4f88e3e8f141084d73ee0565ed21c2ade30019a9df1dc3c3ac749702e4c9742"} Jul 06 19:44:09.316814 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:09.316766 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" podStartSLOduration=1.316751673 podStartE2EDuration="1.316751673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:44:09.313505231 +0000 UTC m=+2276.854227458" watchObservedRunningTime="2026-07-06 19:44:09.316751673 +0000 UTC m=+2276.857473910" Jul 06 19:44:12.160400 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:12.160361 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:44:16.848970 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:16.848934 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:16.950908 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:16.950877 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:18.571970 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:18.571942 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:18.572375 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:18.571992 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:18.583980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:18.583958 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:19.363779 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:19.363746 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:44:25.218773 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:25.218734 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:25.319296 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:25.319272 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:27.163528 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.163501 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2"] Jul 06 19:44:27.167287 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.167266 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.170985 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.170961 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"default-dockercfg-5ln7f\"" Jul 06 19:44:27.170985 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.170970 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"rate-test-model-f31ab4-kserve-self-signed-certs\"" Jul 06 19:44:27.171167 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.170964 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"openshift-service-ca.crt\"" Jul 06 19:44:27.171167 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.171040 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f31ab4\"/\"kube-root-ca.crt\"" Jul 06 19:44:27.175642 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.175620 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2"] Jul 06 19:44:27.248360 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.248334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-tmp-dir\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.248505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.248366 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-model-cache\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.248505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.248407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-dshm\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.248505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.248445 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9897be-6087-4190-b0ce-4c95ecf15af2-tls-certs\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.248505 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.248491 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-home\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.248660 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.248509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjx5s\" (UniqueName: \"kubernetes.io/projected/3d9897be-6087-4190-b0ce-4c95ecf15af2-kube-api-access-zjx5s\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.253829 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.253806 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:27.349133 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9897be-6087-4190-b0ce-4c95ecf15af2-tls-certs\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.349238 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-home\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.349285 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349267 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjx5s\" (UniqueName: \"kubernetes.io/projected/3d9897be-6087-4190-b0ce-4c95ecf15af2-kube-api-access-zjx5s\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.349336 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-tmp-dir\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.349391 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-model-cache\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.349586 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349555 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-home\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.349776 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-dshm\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.349871 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349776 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-model-cache\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.349871 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.349702 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-tmp-dir\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.351531 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.351511 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-dshm\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.351681 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.351665 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9897be-6087-4190-b0ce-4c95ecf15af2-tls-certs\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.357186 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.357163 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjx5s\" (UniqueName: \"kubernetes.io/projected/3d9897be-6087-4190-b0ce-4c95ecf15af2-kube-api-access-zjx5s\") pod \"rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.478681 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.478625 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:27.606900 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:27.606875 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2"] Jul 06 19:44:27.608209 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:44:27.608179 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d9897be_6087_4190_b0ce_4c95ecf15af2.slice/crio-8570c2d2cea6228d4f3a47f1981d8644242c4bad4f8c06dc22d2ce688222c83a WatchSource:0}: Error finding container 8570c2d2cea6228d4f3a47f1981d8644242c4bad4f8c06dc22d2ce688222c83a: Status 404 returned error can't find the container with id 8570c2d2cea6228d4f3a47f1981d8644242c4bad4f8c06dc22d2ce688222c83a Jul 06 19:44:28.402787 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:28.402756 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" event={"ID":"3d9897be-6087-4190-b0ce-4c95ecf15af2","Type":"ContainerStarted","Data":"b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f"} Jul 06 19:44:28.403176 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:28.402796 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" event={"ID":"3d9897be-6087-4190-b0ce-4c95ecf15af2","Type":"ContainerStarted","Data":"8570c2d2cea6228d4f3a47f1981d8644242c4bad4f8c06dc22d2ce688222c83a"} Jul 06 19:44:28.420638 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:28.420598 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" podStartSLOduration=1.42058292 podStartE2EDuration="1.42058292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:44:28.419078604 +0000 UTC m=+2295.959800832" watchObservedRunningTime="2026-07-06 19:44:28.42058292 +0000 UTC m=+2295.961305184" Jul 06 19:44:30.955277 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:30.955242 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:44:37.479598 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:37.479567 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:37.479980 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:37.479611 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:37.492059 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:37.492037 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:38.467850 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:38.467823 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:44:43.314689 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:43.314649 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:43.414523 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:43.414490 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:44:55.372204 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:44:55.372166 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:45:08.855286 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:08.855206 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:45:15.882044 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:15.882005 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg"] Jul 06 19:45:15.902268 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:15.902240 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:16.807410 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:16.807369 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2"] Jul 06 19:45:16.846786 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:16.846722 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:17.903443 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:17.903387 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:18.315470 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:18.315416 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:18.425652 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:18.425626 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:18.523927 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:18.523898 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:18.722771 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:18.722687 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6"] Jul 06 19:45:18.722956 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:18.722927 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" podUID="653eb53c-28ad-4a87-8ad4-04c23037652c" containerName="maas-api" containerID="cri-o://45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c" gracePeriod=30 Jul 06 19:45:18.769016 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:18.768992 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:18.980166 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:18.980117 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:45:19.152699 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.152669 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653eb53c-28ad-4a87-8ad4-04c23037652c-maas-api-tls\") pod \"653eb53c-28ad-4a87-8ad4-04c23037652c\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " Jul 06 19:45:19.152837 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.152758 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrmhp\" (UniqueName: \"kubernetes.io/projected/653eb53c-28ad-4a87-8ad4-04c23037652c-kube-api-access-rrmhp\") pod \"653eb53c-28ad-4a87-8ad4-04c23037652c\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " Jul 06 19:45:19.152837 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.152788 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653eb53c-28ad-4a87-8ad4-04c23037652c-openshift-service-ca\") pod \"653eb53c-28ad-4a87-8ad4-04c23037652c\" (UID: \"653eb53c-28ad-4a87-8ad4-04c23037652c\") " Jul 06 19:45:19.153170 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.153151 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653eb53c-28ad-4a87-8ad4-04c23037652c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "653eb53c-28ad-4a87-8ad4-04c23037652c" (UID: "653eb53c-28ad-4a87-8ad4-04c23037652c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:45:19.154747 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.154727 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653eb53c-28ad-4a87-8ad4-04c23037652c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "653eb53c-28ad-4a87-8ad4-04c23037652c" (UID: "653eb53c-28ad-4a87-8ad4-04c23037652c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:45:19.154824 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.154751 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653eb53c-28ad-4a87-8ad4-04c23037652c-kube-api-access-rrmhp" (OuterVolumeSpecName: "kube-api-access-rrmhp") pod "653eb53c-28ad-4a87-8ad4-04c23037652c" (UID: "653eb53c-28ad-4a87-8ad4-04c23037652c"). InnerVolumeSpecName "kube-api-access-rrmhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:19.254169 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.254115 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrmhp\" (UniqueName: \"kubernetes.io/projected/653eb53c-28ad-4a87-8ad4-04c23037652c-kube-api-access-rrmhp\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:19.254169 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.254136 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653eb53c-28ad-4a87-8ad4-04c23037652c-openshift-service-ca\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:19.254169 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.254145 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653eb53c-28ad-4a87-8ad4-04c23037652c-maas-api-tls\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:19.679055 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.679023 2569 generic.go:356] "Generic (PLEG): container finished" podID="653eb53c-28ad-4a87-8ad4-04c23037652c" containerID="45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c" exitCode=0 Jul 06 19:45:19.679230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.679068 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" Jul 06 19:45:19.679230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.679088 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" event={"ID":"653eb53c-28ad-4a87-8ad4-04c23037652c","Type":"ContainerDied","Data":"45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c"} Jul 06 19:45:19.679230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.679137 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6" event={"ID":"653eb53c-28ad-4a87-8ad4-04c23037652c","Type":"ContainerDied","Data":"21c891e8fae72633ddc223334920305857f0bc9ba2918f9515b0f93b324547a8"} Jul 06 19:45:19.679230 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.679167 2569 scope.go:117] "RemoveContainer" containerID="45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c" Jul 06 19:45:19.688232 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.688217 2569 scope.go:117] "RemoveContainer" containerID="45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c" Jul 06 19:45:19.688506 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:45:19.688484 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c\": container with ID starting with 45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c not found: ID does not exist" containerID="45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c" Jul 06 19:45:19.688575 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.688518 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c"} err="failed to get container status \"45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c\": rpc error: code = NotFound desc = could not find container \"45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c\": container with ID starting with 45864962a09bf74fb66994a001ac20a7a3fde148bec43b8cecbf3aa39d65914c not found: ID does not exist" Jul 06 19:45:19.704209 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.704180 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6"] Jul 06 19:45:19.708111 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:19.708070 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f969f4-64b8778fbd-xh9k6"] Jul 06 19:45:21.021661 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:21.021626 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="653eb53c-28ad-4a87-8ad4-04c23037652c" path="/var/lib/kubelet/pods/653eb53c-28ad-4a87-8ad4-04c23037652c/volumes" Jul 06 19:45:21.859407 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:21.859373 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:45:23.389654 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:23.389624 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:30.314701 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:30.314670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:30.378986 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:30.378958 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:30.906456 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:30.906408 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" podUID="9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" containerName="main" containerID="cri-o://538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6" gracePeriod=45 Jul 06 19:45:31.148362 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.148340 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:45:31.244211 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244135 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-home\") pod \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " Jul 06 19:45:31.244211 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244183 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tmp-dir\") pod \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " Jul 06 19:45:31.244420 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244226 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tls-certs\") pod \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " Jul 06 19:45:31.244420 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244273 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-model-cache\") pod \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " Jul 06 19:45:31.244420 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244322 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn7nq\" (UniqueName: \"kubernetes.io/projected/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-kube-api-access-vn7nq\") pod \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " Jul 06 19:45:31.244420 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244346 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-dshm\") pod \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\" (UID: \"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4\") " Jul 06 19:45:31.244420 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244409 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-home" (OuterVolumeSpecName: "home") pod "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" (UID: "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:31.244648 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244449 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" (UID: "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:31.244648 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244603 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" (UID: "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:31.244750 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244698 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:31.244750 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244718 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:31.244750 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.244731 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:31.246191 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.246168 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" (UID: "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:45:31.246280 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.246239 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-kube-api-access-vn7nq" (OuterVolumeSpecName: "kube-api-access-vn7nq") pod "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" (UID: "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4"). InnerVolumeSpecName "kube-api-access-vn7nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:31.246317 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.246300 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-dshm" (OuterVolumeSpecName: "dshm") pod "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" (UID: "9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:31.345478 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.345457 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:31.345826 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.345480 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7nq\" (UniqueName: \"kubernetes.io/projected/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-kube-api-access-vn7nq\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:31.345826 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.345492 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:31.747182 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.747149 2569 generic.go:356] "Generic (PLEG): container finished" podID="9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" containerID="538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6" exitCode=0 Jul 06 19:45:31.747334 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.747225 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" Jul 06 19:45:31.747334 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.747238 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" event={"ID":"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4","Type":"ContainerDied","Data":"538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6"} Jul 06 19:45:31.747334 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.747288 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg" event={"ID":"9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4","Type":"ContainerDied","Data":"a4f88e3e8f141084d73ee0565ed21c2ade30019a9df1dc3c3ac749702e4c9742"} Jul 06 19:45:31.747334 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.747307 2569 scope.go:117] "RemoveContainer" containerID="538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6" Jul 06 19:45:31.761439 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.761408 2569 scope.go:117] "RemoveContainer" containerID="538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6" Jul 06 19:45:31.761725 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:45:31.761704 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6\": container with ID starting with 538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6 not found: ID does not exist" containerID="538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6" Jul 06 19:45:31.761788 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.761733 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6"} err="failed to get container status \"538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6\": rpc error: code = NotFound desc = could not find container \"538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6\": container with ID starting with 538065e0c80f2d2380743556d51658529a4be0d2486343bea375628414989ad6 not found: ID does not exist" Jul 06 19:45:31.773173 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.773148 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg"] Jul 06 19:45:31.775482 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.775463 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f969f4/rate-test-model-f969f4-kserve-757ff5f68c-8mprg"] Jul 06 19:45:31.826648 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:31.826615 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" podUID="3d9897be-6087-4190-b0ce-4c95ecf15af2" containerName="main" containerID="cri-o://b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f" gracePeriod=45 Jul 06 19:45:32.078552 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.078530 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:45:32.149962 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.149941 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-model-cache\") pod \"3d9897be-6087-4190-b0ce-4c95ecf15af2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " Jul 06 19:45:32.150087 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.149972 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-dshm\") pod \"3d9897be-6087-4190-b0ce-4c95ecf15af2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " Jul 06 19:45:32.150087 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.149989 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9897be-6087-4190-b0ce-4c95ecf15af2-tls-certs\") pod \"3d9897be-6087-4190-b0ce-4c95ecf15af2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " Jul 06 19:45:32.150087 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.150020 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjx5s\" (UniqueName: \"kubernetes.io/projected/3d9897be-6087-4190-b0ce-4c95ecf15af2-kube-api-access-zjx5s\") pod \"3d9897be-6087-4190-b0ce-4c95ecf15af2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " Jul 06 19:45:32.150087 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.150064 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-tmp-dir\") pod \"3d9897be-6087-4190-b0ce-4c95ecf15af2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " Jul 06 19:45:32.150309 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.150097 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-home\") pod \"3d9897be-6087-4190-b0ce-4c95ecf15af2\" (UID: \"3d9897be-6087-4190-b0ce-4c95ecf15af2\") " Jul 06 19:45:32.150309 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.150224 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d9897be-6087-4190-b0ce-4c95ecf15af2" (UID: "3d9897be-6087-4190-b0ce-4c95ecf15af2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:32.150412 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.150302 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d9897be-6087-4190-b0ce-4c95ecf15af2" (UID: "3d9897be-6087-4190-b0ce-4c95ecf15af2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:32.150412 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.150393 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-home" (OuterVolumeSpecName: "home") pod "3d9897be-6087-4190-b0ce-4c95ecf15af2" (UID: "3d9897be-6087-4190-b0ce-4c95ecf15af2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:32.151993 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.151973 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9897be-6087-4190-b0ce-4c95ecf15af2-kube-api-access-zjx5s" (OuterVolumeSpecName: "kube-api-access-zjx5s") pod "3d9897be-6087-4190-b0ce-4c95ecf15af2" (UID: "3d9897be-6087-4190-b0ce-4c95ecf15af2"). InnerVolumeSpecName "kube-api-access-zjx5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:32.152066 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.152019 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9897be-6087-4190-b0ce-4c95ecf15af2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d9897be-6087-4190-b0ce-4c95ecf15af2" (UID: "3d9897be-6087-4190-b0ce-4c95ecf15af2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:45:32.152066 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.152059 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-dshm" (OuterVolumeSpecName: "dshm") pod "3d9897be-6087-4190-b0ce-4c95ecf15af2" (UID: "3d9897be-6087-4190-b0ce-4c95ecf15af2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:32.251241 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.251218 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9897be-6087-4190-b0ce-4c95ecf15af2-tls-certs\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:32.251241 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.251239 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjx5s\" (UniqueName: \"kubernetes.io/projected/3d9897be-6087-4190-b0ce-4c95ecf15af2-kube-api-access-zjx5s\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:32.251364 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.251249 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-tmp-dir\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:32.251364 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.251258 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-home\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:32.251364 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.251265 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-model-cache\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:32.251364 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.251272 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9897be-6087-4190-b0ce-4c95ecf15af2-dshm\") on node \"ip-10-0-131-1.ec2.internal\" DevicePath \"\"" Jul 06 19:45:32.754925 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.754895 2569 generic.go:356] "Generic (PLEG): container finished" podID="3d9897be-6087-4190-b0ce-4c95ecf15af2" containerID="b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f" exitCode=0 Jul 06 19:45:32.755338 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.754944 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" Jul 06 19:45:32.755338 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.754999 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" event={"ID":"3d9897be-6087-4190-b0ce-4c95ecf15af2","Type":"ContainerDied","Data":"b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f"} Jul 06 19:45:32.755338 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.755059 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2" event={"ID":"3d9897be-6087-4190-b0ce-4c95ecf15af2","Type":"ContainerDied","Data":"8570c2d2cea6228d4f3a47f1981d8644242c4bad4f8c06dc22d2ce688222c83a"} Jul 06 19:45:32.755338 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.755087 2569 scope.go:117] "RemoveContainer" containerID="b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f" Jul 06 19:45:32.765395 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.765378 2569 scope.go:117] "RemoveContainer" containerID="b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f" Jul 06 19:45:32.765709 ip-10-0-131-1 kubenswrapper[2569]: E0706 19:45:32.765685 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f\": container with ID starting with b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f not found: ID does not exist" containerID="b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f" Jul 06 19:45:32.765769 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.765717 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f"} err="failed to get container status \"b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f\": rpc error: code = NotFound desc = could not find container \"b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f\": container with ID starting with b38b28899124d46e394bdc376f0f30f765c1a539adf9d732f5d92a2a147deb9f not found: ID does not exist" Jul 06 19:45:32.777799 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.777777 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2"] Jul 06 19:45:32.782337 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:32.782317 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f31ab4/rate-test-model-f31ab4-kserve-78bdf96df5-lnrf2"] Jul 06 19:45:33.023566 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:33.023530 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9897be-6087-4190-b0ce-4c95ecf15af2" path="/var/lib/kubelet/pods/3d9897be-6087-4190-b0ce-4c95ecf15af2/volumes" Jul 06 19:45:33.024005 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:45:33.023983 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" path="/var/lib/kubelet/pods/9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4/volumes" Jul 06 19:46:00.623015 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:00.622983 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:00.899267 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:00.899236 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:01.326145 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:01.326095 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:01.725064 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:01.724990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:01.822503 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:01.822472 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:01.922655 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:01.922632 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:02.051411 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:02.051384 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:04.863943 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:04.863910 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-plldc"] Jul 06 19:46:07.774232 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:07.774201 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:07.843762 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:07.843731 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:08.035909 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:08.035836 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:08.678634 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:08.678606 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-664684f5d4-w9zrl_ece53de8-6b72-4c33-83f4-8dccd52e1c12/authorino/0.log" Jul 06 19:46:13.112327 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:13.112300 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:46:13.124849 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:13.124828 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2lch_7c06366a-b8aa-4ebe-bda8-afcdbbc12f66/ovn-acl-logging/0.log" Jul 06 19:46:13.260272 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:13.260251 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z4lt5_8f7497cb-d60d-48d7-9e8d-b219a58455b5/manager/0.log" Jul 06 19:46:13.381270 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:13.381225 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-br7h7_54fc970a-68fd-45d1-a054-083b681437ea/manager/0.log" Jul 06 19:46:13.636252 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:13.636199 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-852z6_0467ef6a-0bb0-4492-a487-2578ca2b9c9d/server/0.log" Jul 06 19:46:13.750212 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:13.750196 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bsqdc_78ea109b-a990-47ca-a49c-b8737866465c/manager/0.log" Jul 06 19:46:13.869430 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:13.869408 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7598fdd968-9ld5m_e4d03638-f6f0-4d19-8709-06c9e1903f61/copy-manifests/0.log" Jul 06 19:46:13.884823 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:13.884799 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7598fdd968-9ld5m_e4d03638-f6f0-4d19-8709-06c9e1903f61/manager/0.log" Jul 06 19:46:14.891273 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:14.891244 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22_62b6ce61-ff82-4666-80fd-2f55a0ffecbc/util/0.log" Jul 06 19:46:14.898528 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:14.898502 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22_62b6ce61-ff82-4666-80fd-2f55a0ffecbc/pull/0.log" Jul 06 19:46:14.904864 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:14.904842 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22_62b6ce61-ff82-4666-80fd-2f55a0ffecbc/extract/0.log" Jul 06 19:46:15.027199 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.027181 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m_933ceb0d-d79f-4e6f-bcd8-4659996a35ec/util/0.log" Jul 06 19:46:15.034600 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.034565 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m_933ceb0d-d79f-4e6f-bcd8-4659996a35ec/pull/0.log" Jul 06 19:46:15.042910 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.042895 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m_933ceb0d-d79f-4e6f-bcd8-4659996a35ec/extract/0.log" Jul 06 19:46:15.157864 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.157806 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z_8a6da782-ce13-47a7-81ad-e084cb1b9ccb/util/0.log" Jul 06 19:46:15.164674 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.164655 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z_8a6da782-ce13-47a7-81ad-e084cb1b9ccb/pull/0.log" Jul 06 19:46:15.171851 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.171834 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z_8a6da782-ce13-47a7-81ad-e084cb1b9ccb/extract/0.log" Jul 06 19:46:15.278570 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.278533 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:15.280152 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.280134 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q_08c50fa4-be66-43d0-8610-19a7e9111fcc/util/0.log" Jul 06 19:46:15.286365 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.286347 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q_08c50fa4-be66-43d0-8610-19a7e9111fcc/pull/0.log" Jul 06 19:46:15.292705 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.292690 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q_08c50fa4-be66-43d0-8610-19a7e9111fcc/extract/0.log" Jul 06 19:46:15.407433 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.407400 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-664684f5d4-w9zrl_ece53de8-6b72-4c33-83f4-8dccd52e1c12/authorino/0.log" Jul 06 19:46:15.524258 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.524237 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-m2bvx_7621b049-968c-4042-bcce-352cbf155c27/manager/0.log" Jul 06 19:46:15.741713 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.741690 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jtrtn_b5b9dafb-e8e0-46f4-b451-e03dd281dc84/kuadrant-console-plugin/0.log" Jul 06 19:46:15.866736 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.866677 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lkmfh_2fbefc5e-5509-46c2-ab17-fb464a89bc65/registry-server/0.log" Jul 06 19:46:15.984083 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:15.984050 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-djd9q_74fd2182-07f3-4075-a9cc-76c6080d9fa7/manager/0.log" Jul 06 19:46:16.103683 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:16.103659 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-plldc_543b67bc-9f9c-416f-86c5-4c1b391bae04/limitador/0.log" Jul 06 19:46:16.219495 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:16.219447 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ddp6j_82fc05a1-f092-4468-bd99-b9b9a2064399/manager/0.log" Jul 06 19:46:16.815562 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:16.815531 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d54fb4f58-twxjw_aee4644a-c77a-4cbb-9c5b-574c7158ee06/kube-auth-proxy/0.log" Jul 06 19:46:17.894149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:17.894123 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-v77ss_d6d71f96-0ae5-4826-9fe8-c5117bed1e81/main/0.log" Jul 06 19:46:18.245903 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:18.245835 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2vwrs_95581bfd-0ca3-4c28-b8f2-b677e2159ab0/main/0.log" Jul 06 19:46:18.366773 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:18.366750 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-mmtz5_919ccf05-a449-40a8-9f44-c67a6a3214d6/main/0.log" Jul 06 19:46:22.780668 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.780636 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ts2hx/must-gather-5fwtr"] Jul 06 19:46:22.781017 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.780978 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653eb53c-28ad-4a87-8ad4-04c23037652c" containerName="maas-api" Jul 06 19:46:22.781017 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.780988 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="653eb53c-28ad-4a87-8ad4-04c23037652c" containerName="maas-api" Jul 06 19:46:22.781017 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.781009 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" containerName="main" Jul 06 19:46:22.781017 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.781015 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" containerName="main" Jul 06 19:46:22.781149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.781021 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9897be-6087-4190-b0ce-4c95ecf15af2" containerName="main" Jul 06 19:46:22.781149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.781026 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9897be-6087-4190-b0ce-4c95ecf15af2" containerName="main" Jul 06 19:46:22.781149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.781088 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="653eb53c-28ad-4a87-8ad4-04c23037652c" containerName="maas-api" Jul 06 19:46:22.781149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.781094 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9897be-6087-4190-b0ce-4c95ecf15af2" containerName="main" Jul 06 19:46:22.781149 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.781102 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d2c6d4f-bdd2-48c8-b4e8-6c03523aceb4" containerName="main" Jul 06 19:46:22.784253 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.784234 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ts2hx/must-gather-5fwtr" Jul 06 19:46:22.787793 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.787775 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ts2hx\"/\"kube-root-ca.crt\"" Jul 06 19:46:22.787793 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.787791 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ts2hx\"/\"openshift-service-ca.crt\"" Jul 06 19:46:22.787956 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.787775 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ts2hx\"/\"default-dockercfg-pfx6k\"" Jul 06 19:46:22.793285 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.793265 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ts2hx/must-gather-5fwtr"] Jul 06 19:46:22.795509 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.795489 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:22.896456 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.896413 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:22.904627 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.904605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99c3090f-63e3-4534-87e2-775fc6081017-must-gather-output\") pod \"must-gather-5fwtr\" (UID: \"99c3090f-63e3-4534-87e2-775fc6081017\") " pod="openshift-must-gather-ts2hx/must-gather-5fwtr" Jul 06 19:46:22.904715 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:22.904631 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkkh\" (UniqueName: \"kubernetes.io/projected/99c3090f-63e3-4534-87e2-775fc6081017-kube-api-access-jfkkh\") pod \"must-gather-5fwtr\" (UID: \"99c3090f-63e3-4534-87e2-775fc6081017\") " pod="openshift-must-gather-ts2hx/must-gather-5fwtr" Jul 06 19:46:23.005860 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:23.005837 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99c3090f-63e3-4534-87e2-775fc6081017-must-gather-output\") pod \"must-gather-5fwtr\" (UID: \"99c3090f-63e3-4534-87e2-775fc6081017\") " pod="openshift-must-gather-ts2hx/must-gather-5fwtr" Jul 06 19:46:23.005960 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:23.005867 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkkh\" (UniqueName: \"kubernetes.io/projected/99c3090f-63e3-4534-87e2-775fc6081017-kube-api-access-jfkkh\") pod \"must-gather-5fwtr\" (UID: \"99c3090f-63e3-4534-87e2-775fc6081017\") " pod="openshift-must-gather-ts2hx/must-gather-5fwtr" Jul 06 19:46:23.006144 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:23.006126 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/99c3090f-63e3-4534-87e2-775fc6081017-must-gather-output\") pod \"must-gather-5fwtr\" (UID: \"99c3090f-63e3-4534-87e2-775fc6081017\") " pod="openshift-must-gather-ts2hx/must-gather-5fwtr" Jul 06 19:46:23.014828 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:23.014798 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkkh\" (UniqueName: \"kubernetes.io/projected/99c3090f-63e3-4534-87e2-775fc6081017-kube-api-access-jfkkh\") pod \"must-gather-5fwtr\" (UID: \"99c3090f-63e3-4534-87e2-775fc6081017\") " pod="openshift-must-gather-ts2hx/must-gather-5fwtr" Jul 06 19:46:23.093544 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:23.093488 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ts2hx/must-gather-5fwtr" Jul 06 19:46:23.222724 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:23.222677 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ts2hx/must-gather-5fwtr"] Jul 06 19:46:23.224934 ip-10-0-131-1 kubenswrapper[2569]: W0706 19:46:23.224906 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99c3090f_63e3_4534_87e2_775fc6081017.slice/crio-2d5b9b75d96d6c0220dcd63c0ce9f7421074b924d6e715e7dc022cc3cf1aaec1 WatchSource:0}: Error finding container 2d5b9b75d96d6c0220dcd63c0ce9f7421074b924d6e715e7dc022cc3cf1aaec1: Status 404 returned error can't find the container with id 2d5b9b75d96d6c0220dcd63c0ce9f7421074b924d6e715e7dc022cc3cf1aaec1 Jul 06 19:46:23.226822 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:23.226806 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:46:24.042936 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:24.042895 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ts2hx/must-gather-5fwtr" event={"ID":"99c3090f-63e3-4534-87e2-775fc6081017","Type":"ContainerStarted","Data":"2d5b9b75d96d6c0220dcd63c0ce9f7421074b924d6e715e7dc022cc3cf1aaec1"} Jul 06 19:46:25.052766 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:25.052729 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ts2hx/must-gather-5fwtr" event={"ID":"99c3090f-63e3-4534-87e2-775fc6081017","Type":"ContainerStarted","Data":"494954a6fbbfed4b6e48afc268f6c7761ff8ff5606c92945fb846674e2e52a5e"} Jul 06 19:46:25.052766 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:25.052771 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ts2hx/must-gather-5fwtr" event={"ID":"99c3090f-63e3-4534-87e2-775fc6081017","Type":"ContainerStarted","Data":"eeacc52b7dd6b16dc940ebe37b8358a7c5b977ec66701543a72e6cd584de24bf"} Jul 06 19:46:25.072141 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:25.072091 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ts2hx/must-gather-5fwtr" podStartSLOduration=2.162064923 podStartE2EDuration="3.072073571s" totalImagesPullingTime="910.008648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:46:25.069413765 +0000 UTC m=+2412.610135995" watchObservedRunningTime="2026-07-06 19:46:25.072073571 +0000 UTC m=+2412.612795799" Jul 06 19:46:25.753809 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:25.753763 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fmm9q_09a32004-05e2-4a14-9943-7b9b3229cf2f/global-pull-secret-syncer/0.log" Jul 06 19:46:25.852295 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:25.852260 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" Jul 06 19:46:25.944524 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:25.944480 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-1.ec2.internal_1eb58fae0b7cc9453d7a3dd0280f4dd9/haproxy/0.log" Jul 06 19:46:26.787398 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:26.787367 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:26.848143 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:26.848074 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cf7h9_6ad6fd62-c4cf-47c3-a1aa-4d71e1fe245e/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:29.690935 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.690906 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22_62b6ce61-ff82-4666-80fd-2f55a0ffecbc/extract/0.log" Jul 06 19:46:29.717290 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.717217 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22_62b6ce61-ff82-4666-80fd-2f55a0ffecbc/util/0.log" Jul 06 19:46:29.741932 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.741881 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759s7s22_62b6ce61-ff82-4666-80fd-2f55a0ffecbc/pull/0.log" Jul 06 19:46:29.781487 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.781451 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m_933ceb0d-d79f-4e6f-bcd8-4659996a35ec/extract/0.log" Jul 06 19:46:29.805516 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.805351 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m_933ceb0d-d79f-4e6f-bcd8-4659996a35ec/util/0.log" Jul 06 19:46:29.836899 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.836822 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bpj4m_933ceb0d-d79f-4e6f-bcd8-4659996a35ec/pull/0.log" Jul 06 19:46:29.879310 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.879264 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z_8a6da782-ce13-47a7-81ad-e084cb1b9ccb/extract/0.log" Jul 06 19:46:29.907620 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.907597 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z_8a6da782-ce13-47a7-81ad-e084cb1b9ccb/util/0.log" Jul 06 19:46:29.938128 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.938102 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tws6z_8a6da782-ce13-47a7-81ad-e084cb1b9ccb/pull/0.log" Jul 06 19:46:29.973337 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:29.973241 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q_08c50fa4-be66-43d0-8610-19a7e9111fcc/extract/0.log" Jul 06 19:46:30.012622 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.012597 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q_08c50fa4-be66-43d0-8610-19a7e9111fcc/util/0.log" Jul 06 19:46:30.066457 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.066378 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vbn4q_08c50fa4-be66-43d0-8610-19a7e9111fcc/pull/0.log" Jul 06 19:46:30.432111 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.432078 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-664684f5d4-w9zrl_ece53de8-6b72-4c33-83f4-8dccd52e1c12/authorino/0.log" Jul 06 19:46:30.480198 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.480168 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-m2bvx_7621b049-968c-4042-bcce-352cbf155c27/manager/0.log" Jul 06 19:46:30.548005 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.547977 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jtrtn_b5b9dafb-e8e0-46f4-b451-e03dd281dc84/kuadrant-console-plugin/0.log" Jul 06 19:46:30.634849 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.634821 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lkmfh_2fbefc5e-5509-46c2-ab17-fb464a89bc65/registry-server/0.log" Jul 06 19:46:30.779447 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.779354 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-djd9q_74fd2182-07f3-4075-a9cc-76c6080d9fa7/manager/0.log" Jul 06 19:46:30.844189 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.844017 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-plldc_543b67bc-9f9c-416f-86c5-4c1b391bae04/limitador/0.log" Jul 06 19:46:30.971564 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:30.971533 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ddp6j_82fc05a1-f092-4468-bd99-b9b9a2064399/manager/0.log" Jul 06 19:46:33.035666 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.035633 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-96bl2_a8d5de75-d3ef-43be-8930-b2af5823b6ab/monitoring-plugin/0.log" Jul 06 19:46:33.148044 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.148009 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-szsdw_2382d583-1001-4a8d-8ed6-bffd91e5c90f/node-exporter/0.log" Jul 06 19:46:33.172024 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.171992 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-szsdw_2382d583-1001-4a8d-8ed6-bffd91e5c90f/kube-rbac-proxy/0.log" Jul 06 19:46:33.196788 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.196758 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-szsdw_2382d583-1001-4a8d-8ed6-bffd91e5c90f/init-textfile/0.log" Jul 06 19:46:33.316166 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.316033 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rbq6k_9a3ed9b2-92e1-4985-a593-ccdab305114b/kube-rbac-proxy-main/0.log" Jul 06 19:46:33.339662 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.339544 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rbq6k_9a3ed9b2-92e1-4985-a593-ccdab305114b/kube-rbac-proxy-self/0.log" Jul 06 19:46:33.367109 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.367077 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rbq6k_9a3ed9b2-92e1-4985-a593-ccdab305114b/openshift-state-metrics/0.log" Jul 06 19:46:33.643251 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.643152 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-bhrps_297fdc92-dd2d-45e8-81ee-8fb348b0b1c6/prometheus-operator/0.log" Jul 06 19:46:33.662591 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.662563 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-bhrps_297fdc92-dd2d-45e8-81ee-8fb348b0b1c6/kube-rbac-proxy/0.log" Jul 06 19:46:33.694191 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.694159 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-spplt_9bc9d731-ab5c-45eb-96e0-82cd518ba6df/prometheus-operator-admission-webhook/0.log" Jul 06 19:46:33.729482 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.729418 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-859499568-8r4sz_ad8bfb29-5f89-495a-a3db-5962a3e89124/telemeter-client/0.log" Jul 06 19:46:33.753177 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.753143 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-859499568-8r4sz_ad8bfb29-5f89-495a-a3db-5962a3e89124/reload/0.log" Jul 06 19:46:33.777300 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.777277 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-859499568-8r4sz_ad8bfb29-5f89-495a-a3db-5962a3e89124/kube-rbac-proxy/0.log" Jul 06 19:46:33.812255 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.812221 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799df989cd-smgpk_43a9777c-503c-4206-b4b6-768a95a28bc9/thanos-query/0.log" Jul 06 19:46:33.836811 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.836774 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799df989cd-smgpk_43a9777c-503c-4206-b4b6-768a95a28bc9/kube-rbac-proxy-web/0.log" Jul 06 19:46:33.863827 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.863799 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799df989cd-smgpk_43a9777c-503c-4206-b4b6-768a95a28bc9/kube-rbac-proxy/0.log" Jul 06 19:46:33.891147 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.891117 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799df989cd-smgpk_43a9777c-503c-4206-b4b6-768a95a28bc9/prom-label-proxy/0.log" Jul 06 19:46:33.918807 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.918737 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799df989cd-smgpk_43a9777c-503c-4206-b4b6-768a95a28bc9/kube-rbac-proxy-rules/0.log" Jul 06 19:46:33.961078 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:33.961052 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799df989cd-smgpk_43a9777c-503c-4206-b4b6-768a95a28bc9/kube-rbac-proxy-metrics/0.log" Jul 06 19:46:35.370482 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.370380 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b"] Jul 06 19:46:35.376717 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.376690 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.384859 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.384832 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b"] Jul 06 19:46:35.521452 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.521131 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-lib-modules\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.521452 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.521208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdz5l\" (UniqueName: \"kubernetes.io/projected/f6673a7c-1f11-407e-961d-de430a0af8e1-kube-api-access-jdz5l\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.521452 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.521251 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-podres\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.521452 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.521305 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-proc\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.521452 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.521344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-sys\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.622611 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-sys\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.622611 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622584 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-lib-modules\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.622852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622633 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdz5l\" (UniqueName: \"kubernetes.io/projected/f6673a7c-1f11-407e-961d-de430a0af8e1-kube-api-access-jdz5l\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.622852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622671 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-podres\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.622852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622669 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-sys\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.622852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622800 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-podres\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.622852 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622805 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-proc\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.623049 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622799 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-lib-modules\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.623049 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.622899 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6673a7c-1f11-407e-961d-de430a0af8e1-proc\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.632676 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.632629 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdz5l\" (UniqueName: \"kubernetes.io/projected/f6673a7c-1f11-407e-961d-de430a0af8e1-kube-api-access-jdz5l\") pod \"perf-node-gather-daemonset-5kd9b\" (UID: \"f6673a7c-1f11-407e-961d-de430a0af8e1\") " pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.691013 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.690982 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:35.875884 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:35.875832 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b"] Jul 06 19:46:36.165571 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:36.165475 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" event={"ID":"f6673a7c-1f11-407e-961d-de430a0af8e1","Type":"ContainerStarted","Data":"6b49058aef8f783719a236548139c53bedecb37a0914b9e7ffde5acd50c909f2"} Jul 06 19:46:36.165744 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:36.165580 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" event={"ID":"f6673a7c-1f11-407e-961d-de430a0af8e1","Type":"ContainerStarted","Data":"f5d5f0d9c4b9db68f1d0df4a324af882b2d2fd08a6b68fa6bc4557bc7e0aa0c9"} Jul 06 19:46:36.166355 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:36.166327 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:36.188032 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:36.187985 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" podStartSLOduration=1.187971084 podStartE2EDuration="1.187971084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:46:36.183638946 +0000 UTC m=+2423.724361192" watchObservedRunningTime="2026-07-06 19:46:36.187971084 +0000 UTC m=+2423.728693312" Jul 06 19:46:38.148973 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:38.148945 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kt26b_36ecd44d-1d0b-499b-9a05-a017cb70fa44/dns/0.log" Jul 06 19:46:38.171585 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:38.171560 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kt26b_36ecd44d-1d0b-499b-9a05-a017cb70fa44/kube-rbac-proxy/0.log" Jul 06 19:46:38.200466 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:38.200441 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d9c9k_17ec7281-57ce-453e-80be-77e12bca6d52/dns-node-resolver/0.log" Jul 06 19:46:38.932724 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:38.932698 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v9t48_5f669448-6e2c-46f1-9e91-8ff68e9886a0/node-ca/0.log" Jul 06 19:46:40.151888 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:40.151858 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d54fb4f58-twxjw_aee4644a-c77a-4cbb-9c5b-574c7158ee06/kube-auth-proxy/0.log" Jul 06 19:46:41.025272 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:41.025248 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-99ktz_10e66372-e151-4119-91da-bfdf10300f53/serve-healthcheck-canary/0.log" Jul 06 19:46:41.602970 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:41.602942 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vbxq_c4c8c169-88e1-4115-8387-7107d15cf4ca/kube-rbac-proxy/0.log" Jul 06 19:46:41.626795 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:41.626768 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vbxq_c4c8c169-88e1-4115-8387-7107d15cf4ca/exporter/0.log" Jul 06 19:46:41.649331 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:41.649309 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vbxq_c4c8c169-88e1-4115-8387-7107d15cf4ca/extractor/0.log" Jul 06 19:46:43.186722 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:43.186694 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ts2hx/perf-node-gather-daemonset-5kd9b" Jul 06 19:46:44.003508 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:44.003484 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z4lt5_8f7497cb-d60d-48d7-9e8d-b219a58455b5/manager/0.log" Jul 06 19:46:44.180908 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:44.180814 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-br7h7_54fc970a-68fd-45d1-a054-083b681437ea/manager/0.log" Jul 06 19:46:44.339392 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:44.339358 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-852z6_0467ef6a-0bb0-4492-a487-2578ca2b9c9d/server/0.log" Jul 06 19:46:44.433307 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:44.433229 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bsqdc_78ea109b-a990-47ca-a49c-b8737866465c/manager/0.log" Jul 06 19:46:44.526207 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:44.526178 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7598fdd968-9ld5m_e4d03638-f6f0-4d19-8709-06c9e1903f61/manager/0.log" Jul 06 19:46:44.554885 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:44.554859 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7598fdd968-9ld5m_e4d03638-f6f0-4d19-8709-06c9e1903f61/copy-manifests/0.log" Jul 06 19:46:45.463906 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:45.463876 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d54fb4f58-twxjw_aee4644a-c77a-4cbb-9c5b-574c7158ee06/kube-auth-proxy/0.log" Jul 06 19:46:46.937239 ip-10-0-131-1 kubenswrapper[2569]: I0706 19:46:46.937167 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d54fb4f58-twxjw_aee4644a-c77a-4cbb-9c5b-574c7158ee06/kube-auth-proxy/0.log"