Jul 06 19:24:01.339792 ip-10-0-136-156 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 19:24:01.339806 ip-10-0-136-156 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 19:24:01.339816 ip-10-0-136-156 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 19:24:01.340135 ip-10-0-136-156 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 19:24:11.072557 ip-10-0-136-156 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 19:24:11.072574 ip-10-0-136-156 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 81b320948c2747c3b32a4174011f3603 -- Jul 06 19:25:59.008713 ip-10-0-136-156 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:25:59.408876 ip-10-0-136-156 kubenswrapper[2545]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:25:59.408876 ip-10-0-136-156 kubenswrapper[2545]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:25:59.408876 ip-10-0-136-156 kubenswrapper[2545]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:25:59.408876 ip-10-0-136-156 kubenswrapper[2545]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:25:59.408876 ip-10-0-136-156 kubenswrapper[2545]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:25:59.409961 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.409880 2545 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:25:59.415279 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415262 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:25:59.415279 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415276 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:25:59.415279 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415280 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:25:59.415279 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415284 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415287 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415290 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415294 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415297 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415300 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415303 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415305 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415308 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415311 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415313 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415316 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415319 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415322 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415324 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415327 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415330 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415333 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415335 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415338 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:25:59.415438 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415341 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415343 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415346 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415348 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415351 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415353 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415356 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415358 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415360 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415363 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415365 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415368 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415370 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415372 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415376 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415379 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415382 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415384 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415387 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415390 2545 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:25:59.415909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415392 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415395 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415397 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415400 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415403 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415405 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415408 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415410 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415427 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415430 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415434 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415438 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415441 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415444 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415446 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415449 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415451 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415454 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415456 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:25:59.416389 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415459 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415462 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415464 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415467 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415469 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415472 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415474 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415478 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415481 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415484 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415486 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415491 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415495 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415498 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415500 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415503 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415506 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415508 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415511 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415513 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:25:59.416858 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415516 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415518 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415521 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415523 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415526 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415528 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415531 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415533 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415536 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415538 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415541 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415543 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415547 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415550 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415553 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415555 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.415558 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.416973 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.416985 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:25:59.417333 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.416989 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.416993 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.416996 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.416999 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417001 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417004 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417006 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417009 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417012 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417014 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417017 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417020 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417022 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417025 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417027 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417031 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417035 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417037 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417040 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417043 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:25:59.417838 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417045 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417047 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417050 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417053 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417055 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417058 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417060 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417063 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417065 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417067 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417070 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417074 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417077 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417079 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417082 2545 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417084 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417087 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417089 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417092 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417094 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:25:59.418323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417096 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417098 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417101 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417103 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417105 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417109 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417111 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417113 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417115 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417119 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417122 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417125 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417128 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417131 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417133 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417136 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417139 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417141 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:25:59.418839 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417145 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417148 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417151 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417155 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417157 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417160 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417163 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417165 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417168 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417170 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417172 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417175 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417177 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417180 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417182 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417184 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417187 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417189 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417193 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417195 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:25:59.419270 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417198 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417200 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417203 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417205 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417207 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417210 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417212 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417215 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417217 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417220 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417222 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417225 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417228 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417230 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417234 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417239 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417242 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417245 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417248 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:25:59.419780 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417625 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417630 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417633 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417636 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417639 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417641 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417644 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417646 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417649 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417651 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417654 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417656 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417659 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417661 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417664 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417666 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417669 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417671 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417674 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:25:59.420345 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417678 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417681 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417684 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417686 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417689 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417693 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417696 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417698 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417701 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417704 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417707 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417711 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417714 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417718 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417720 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417723 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417725 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417727 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417730 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:25:59.420817 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417732 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417735 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417738 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417741 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417743 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417746 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417748 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417751 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417753 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417756 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417758 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417761 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417764 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417766 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417769 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417771 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417773 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417776 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417779 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417781 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:25:59.421275 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417784 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417786 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417789 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417792 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417794 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417796 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417799 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417802 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417805 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417807 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417810 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417812 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417814 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417817 2545 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417819 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417821 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417824 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417826 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417828 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:25:59.421768 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417831 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417833 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417836 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417838 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417840 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417843 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417846 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417848 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417851 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417853 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417856 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417860 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417863 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417865 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417868 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417871 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417873 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417876 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417880 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417883 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:25:59.422242 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417886 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.417889 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417954 2545 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417961 2545 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417966 2545 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417970 2545 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417974 2545 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417977 2545 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417980 2545 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417984 2545 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417987 2545 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417990 2545 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417993 2545 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.417996 2545 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418000 2545 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418003 2545 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418005 2545 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418008 2545 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418011 2545 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418013 2545 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418017 2545 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418020 2545 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418023 2545 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418028 2545 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:25:59.422820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418031 2545 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418034 2545 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418037 2545 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418040 2545 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418043 2545 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418046 2545 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418049 2545 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418052 2545 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418055 2545 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418057 2545 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418062 2545 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418065 2545 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418067 2545 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418070 2545 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418073 2545 flags.go:64] FLAG: --enable-server="true" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418076 2545 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418081 2545 flags.go:64] FLAG: --event-burst="100" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418084 2545 flags.go:64] FLAG: --event-qps="50" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418087 2545 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418089 2545 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418092 2545 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418096 2545 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418098 2545 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418101 2545 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418104 2545 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:25:59.423381 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418107 2545 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418109 2545 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418112 2545 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418115 2545 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418118 2545 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418120 2545 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418124 2545 flags.go:64] FLAG: --feature-gates="" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418128 2545 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418130 2545 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418133 2545 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418136 2545 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418139 2545 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418142 2545 flags.go:64] FLAG: --help="false" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418145 2545 flags.go:64] FLAG: --hostname-override="ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418148 2545 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418150 2545 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418153 2545 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418156 2545 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418160 2545 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418163 2545 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418166 2545 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418168 2545 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418171 2545 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:25:59.424003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418174 2545 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418177 2545 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418179 2545 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418182 2545 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418184 2545 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418187 2545 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418190 2545 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418193 2545 flags.go:64] FLAG: --lock-file="" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418195 2545 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418198 2545 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418201 2545 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418206 2545 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418208 2545 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418211 2545 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418214 2545 flags.go:64] FLAG: --logging-format="text" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418218 2545 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418221 2545 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418224 2545 flags.go:64] FLAG: --manifest-url="" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418226 2545 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418232 2545 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418235 2545 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418238 2545 flags.go:64] FLAG: --max-pods="110" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418241 2545 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418244 2545 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418247 2545 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:25:59.424637 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418249 2545 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418252 2545 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418255 2545 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418258 2545 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418265 2545 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418267 2545 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418270 2545 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418273 2545 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418276 2545 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418281 2545 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418283 2545 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418286 2545 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418289 2545 flags.go:64] FLAG: --port="10250" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418292 2545 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418295 2545 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f123903895a1406" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418298 2545 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418300 2545 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418303 2545 flags.go:64] FLAG: --register-node="true" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418306 2545 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418309 2545 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418312 2545 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418314 2545 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418317 2545 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418324 2545 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418327 2545 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:25:59.425237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418329 2545 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418333 2545 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418336 2545 flags.go:64] FLAG: --runonce="false" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418339 2545 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418342 2545 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418345 2545 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418347 2545 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418350 2545 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418353 2545 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418356 2545 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418359 2545 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418362 2545 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418365 2545 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418368 2545 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418371 2545 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418373 2545 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418376 2545 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418378 2545 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418384 2545 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418386 2545 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418389 2545 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418393 2545 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418395 2545 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418398 2545 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418400 2545 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:25:59.426143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418403 2545 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418405 2545 flags.go:64] FLAG: --v="2" Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418409 2545 flags.go:64] FLAG: --version="false" Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418429 2545 flags.go:64] FLAG: --vmodule="" Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418433 2545 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418438 2545 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418556 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418560 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418563 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418566 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418569 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418571 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418574 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418576 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418579 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418581 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418584 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418587 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418589 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418592 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418595 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:25:59.426969 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418597 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418599 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418602 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418604 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418607 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418609 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418611 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418614 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418616 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418619 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418621 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418623 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418626 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418628 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418631 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418633 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418637 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418640 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418642 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418645 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:25:59.427501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418647 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418650 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418652 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418655 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418657 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418660 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418662 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418665 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418667 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418670 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418672 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418675 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418677 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418680 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418682 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418685 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418687 2545 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418689 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418692 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418694 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:25:59.427996 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418696 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418699 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418701 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418704 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418706 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418710 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418714 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418717 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418721 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418724 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418726 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418729 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418732 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418734 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418737 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418740 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418744 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418747 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418750 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:25:59.428576 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418753 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418756 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418760 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418763 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418765 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418768 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418770 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418773 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418776 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418780 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418783 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418785 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418788 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418790 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418792 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418795 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418797 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418799 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418802 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:25:59.429036 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418804 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418807 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418811 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418814 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418816 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.418819 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.418824 2545 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.427263 2545 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.427280 2545 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.428029 2545 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.428040 2545 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428096 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428101 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428104 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:25:59.429501 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428107 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428110 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428113 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428115 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428117 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428120 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428123 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428125 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428128 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428131 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428133 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428135 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428138 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428140 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428143 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428145 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428148 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428150 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428153 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428155 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:25:59.429844 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428158 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428161 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428163 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428166 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428168 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428170 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428174 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428176 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428179 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428182 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428184 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428186 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428190 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428194 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428197 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428200 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428203 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428206 2545 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428209 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:25:59.430327 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428213 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428216 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428218 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428221 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428223 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428226 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428229 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428231 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428233 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428236 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428238 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428241 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428243 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428246 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428248 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428251 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428253 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428256 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428258 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428261 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:25:59.430790 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428264 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428266 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428269 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428272 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428274 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428277 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428279 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428282 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428284 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428287 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428289 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428291 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428294 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428296 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428299 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428301 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428304 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428307 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428310 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428312 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:25:59.431264 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428315 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428317 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428319 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428322 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428325 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428327 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428330 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428332 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428335 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428338 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428340 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428343 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428346 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428349 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428352 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428356 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:25:59.431767 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428358 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.428364 2545 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428484 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428489 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428492 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428494 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428497 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428501 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428504 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428507 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428510 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428512 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428515 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428517 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428520 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:25:59.432183 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428523 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428526 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428528 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428531 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428534 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428537 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428539 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428542 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428544 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428546 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428549 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428552 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428554 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428557 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428559 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428562 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428564 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428567 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428569 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428571 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:25:59.432621 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428574 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428576 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428579 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428582 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428584 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428587 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428589 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428592 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428594 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428596 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428599 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428601 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428604 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428607 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428609 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428612 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428614 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428616 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428619 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428621 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:25:59.433105 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428623 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428626 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428628 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428632 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428636 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428640 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428643 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428646 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428649 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428652 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428655 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428657 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428660 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428662 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428665 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428667 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428670 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428672 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428674 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:25:59.433592 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428677 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428679 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428682 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428684 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428686 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428689 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428692 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428695 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428697 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428699 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428701 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428704 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428706 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428708 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428711 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428713 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428716 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428719 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428721 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:25:59.434050 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428723 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428726 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428728 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428730 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428733 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428735 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428738 2545 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:25:59.428740 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.428745 2545 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.428862 2545 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.431218 2545 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.432847 2545 server.go:1019] "Starting client certificate rotation" Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.432946 2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:25:59.434520 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.432983 2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:25:59.435450 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.435438 2545 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:25:59.436695 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.436683 2545 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:25:59.449558 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.449541 2545 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:25:59.452137 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.452119 2545 log.go:25] "Validated CRI v1 image API" Jul 06 19:25:59.452924 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.452912 2545 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:25:59.456073 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.456050 2545 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e7f2824d-e88f-494e-a57a-c6416ddba12b:/dev/nvme0n1p4 efa8365b-958f-454f-a110-b4d386bf42da:/dev/nvme0n1p3] Jul 06 19:25:59.456121 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.456074 2545 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:25:59.460236 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.460221 2545 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:25:59.461603 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.461505 2545 manager.go:216] Machine: {Timestamp:2026-07-06 19:25:59.460187381 +0000 UTC m=+0.335734054 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100237 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23fb1cdb6a83dd8a71e7070e11b614 SystemUUID:ec23fb1c-db6a-83dd-8a71-e7070e11b614 BootID:81b32094-8c27-47c3-b32a-4174011f3603 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:02:73:bf:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:02:73:bf:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:7a:9e:ca:db:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:25:59.461603 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.461602 2545 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:25:59.461711 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.461698 2545 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:25:59.466541 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.466513 2545 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:25:59.466695 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.466544 2545 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:25:59.467264 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.467253 2545 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:25:59.467293 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.467267 2545 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:25:59.467293 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.467274 2545 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:25:59.467876 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.467865 2545 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:25:59.469681 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.469671 2545 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:25:59.472960 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.472949 2545 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:25:59.473068 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.473059 2545 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:25:59.473097 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.473087 2545 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:25:59.473097 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.473096 2545 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:25:59.473146 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.473109 2545 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:25:59.473146 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.473118 2545 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:25:59.473146 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.473126 2545 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:25:59.473489 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.473476 2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:25:59.473522 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.473504 2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:25:59.476063 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.476050 2545 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:25:59.478304 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.478291 2545 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:25:59.478345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.478322 2545 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:25:59.479549 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479536 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:25:59.479590 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479557 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:25:59.479590 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479563 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:25:59.479590 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479568 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:25:59.479590 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479577 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:25:59.479590 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479582 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:25:59.479590 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479590 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:25:59.479739 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479595 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:25:59.479739 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479601 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:25:59.479739 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479606 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:25:59.479739 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479614 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:25:59.479739 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.479619 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:25:59.481193 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.481182 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:25:59.481224 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.481194 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:25:59.484643 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.484632 2545 server.go:1268] "Started kubelet" Jul 06 19:25:59.484781 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.484736 2545 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:25:59.484864 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.484850 2545 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:25:59.484912 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.484732 2545 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:25:59.485481 ip-10-0-136-156 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:25:59.485616 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.485519 2545 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:25:59.485616 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.485528 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:25:59.485616 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.485591 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:25:59.487638 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.487613 2545 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:25:59.489060 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.489036 2545 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:25:59.490448 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.490407 2545 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:25:59.495662 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.494630 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18bfc9e4379a7ea3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:ip-10-0-136-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-07-06 19:25:59.484604067 +0000 UTC m=+0.360150738,LastTimestamp:2026-07-06 19:25:59.484604067 +0000 UTC m=+0.360150738,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Jul 06 19:25:59.496452 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.496434 2545 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:25:59.496527 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.496433 2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:25:59.496570 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.496545 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:25:59.497280 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497237 2545 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:25:59.497280 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497240 2545 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:25:59.497408 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497290 2545 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:25:59.497408 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497322 2545 factory.go:55] Registering systemd factory Jul 06 19:25:59.497408 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497337 2545 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:25:59.497558 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497477 2545 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:25:59.497558 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497487 2545 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:25:59.497615 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497561 2545 factory.go:153] Registering CRI-O factory Jul 06 19:25:59.497615 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497575 2545 factory.go:223] Registration of the crio container factory successfully Jul 06 19:25:59.497700 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497646 2545 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:25:59.497700 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497667 2545 factory.go:103] Registering Raw factory Jul 06 19:25:59.497700 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.497680 2545 manager.go:1196] Started watching for new ooms in manager Jul 06 19:25:59.498219 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.498206 2545 manager.go:349] Starting recovery of all containers Jul 06 19:25:59.509097 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.508936 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:25:59.509295 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.509274 2545 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:25:59.510574 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.510554 2545 manager.go:354] Recovery completed Jul 06 19:25:59.511971 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.511954 2545 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 19:25:59.514944 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.514932 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:25:59.516919 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.516898 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:25:59.516968 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.516932 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:25:59.516968 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.516941 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:25:59.517451 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.517437 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:25:59.517500 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.517452 2545 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:25:59.517500 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.517462 2545 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:25:59.517500 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.517480 2545 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:25:59.518132 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.518119 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:25:59.518185 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.518175 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:25:59.518219 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.518190 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:25:59.518827 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.518814 2545 policy_none.go:49] "None policy: Start" Jul 06 19:25:59.518868 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.518835 2545 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:25:59.518868 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.518849 2545 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:25:59.518946 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.518871 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18bfc9e439879810 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:ip-10-0-136-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-07-06 19:25:59.516919824 +0000 UTC m=+0.392466499,LastTimestamp:2026-07-06 19:25:59.516919824 +0000 UTC m=+0.392466499,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Jul 06 19:25:59.519747 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.519736 2545 policy_none.go:47] "Start" Jul 06 19:25:59.525720 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.525705 2545 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zqws6" Jul 06 19:25:59.528275 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.528205 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18bfc9e43987d86d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:ip-10-0-136-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-156.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-07-06 19:25:59.516936301 +0000 UTC m=+0.392482976,LastTimestamp:2026-07-06 19:25:59.516936301 +0000 UTC m=+0.392482976,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Jul 06 19:25:59.534795 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.534777 2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zqws6" Jul 06 19:25:59.553951 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.553936 2545 manager.go:337] "Starting Device Plugin manager" Jul 06 19:25:59.554029 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.553969 2545 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:25:59.554029 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.553981 2545 server.go:87] "Starting device plugin registration server" Jul 06 19:25:59.554204 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.554191 2545 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:25:59.554271 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.554205 2545 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:25:59.554318 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.554282 2545 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:25:59.554382 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.554369 2545 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:25:59.554382 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.554381 2545 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:25:59.554844 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.554823 2545 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:25:59.554921 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.554878 2545 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:25:59.596479 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.596460 2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:25:59.597744 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.597728 2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:25:59.597817 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.597765 2545 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:25:59.597817 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.597785 2545 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:25:59.597901 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.597823 2545 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:25:59.599761 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.599744 2545 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:25:59.654702 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.654681 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:25:59.655646 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.655633 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:25:59.655707 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.655661 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:25:59.655707 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.655670 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:25:59.655707 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.655690 2545 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.664897 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.664859 2545 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.664897 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.664879 2545 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-156.ec2.internal\": node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:25:59.678335 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.678316 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:25:59.698723 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.698704 2545 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal"] Jul 06 19:25:59.698774 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.698766 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:25:59.699473 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.699459 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:25:59.699523 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.699486 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:25:59.699523 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.699501 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:25:59.700648 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.700637 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:25:59.700739 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.700726 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.700775 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.700752 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:25:59.701312 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.701296 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:25:59.701377 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.701327 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:25:59.701377 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.701337 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:25:59.701377 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.701300 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:25:59.701537 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.701381 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:25:59.701537 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.701397 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:25:59.702666 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.702651 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.702714 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.702675 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:25:59.703274 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.703259 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:25:59.703345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.703288 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:25:59.703345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.703301 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:25:59.731053 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.731035 2545 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.735219 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.735204 2545 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.779440 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.779406 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:25:59.799398 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.799378 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623f264d6030616a9f10578731099164-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"623f264d6030616a9f10578731099164\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.799472 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.799406 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623f264d6030616a9f10578731099164-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"623f264d6030616a9f10578731099164\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.799472 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.799446 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceedd80492377588ff46c4ba5374d82e-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"ceedd80492377588ff46c4ba5374d82e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.880185 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.880162 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:25:59.900542 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.900523 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623f264d6030616a9f10578731099164-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"623f264d6030616a9f10578731099164\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.900592 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.900547 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623f264d6030616a9f10578731099164-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"623f264d6030616a9f10578731099164\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.900592 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.900563 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceedd80492377588ff46c4ba5374d82e-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"ceedd80492377588ff46c4ba5374d82e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.900592 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.900588 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceedd80492377588ff46c4ba5374d82e-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"ceedd80492377588ff46c4ba5374d82e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.900681 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.900609 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623f264d6030616a9f10578731099164-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"623f264d6030616a9f10578731099164\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.900681 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:25:59.900613 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623f264d6030616a9f10578731099164-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"623f264d6030616a9f10578731099164\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:25:59.980957 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:25:59.980898 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:26:00.034450 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.034408 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:26:00.038344 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.038282 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 06 19:26:00.081949 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:00.081925 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:26:00.182493 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:00.182470 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:26:00.283016 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:00.282963 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:26:00.383552 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:00.383526 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:26:00.431931 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.431916 2545 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:26:00.432392 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.432051 2545 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:26:00.484619 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:00.484590 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:26:00.528901 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:00.528876 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceedd80492377588ff46c4ba5374d82e.slice/crio-63013c3bc74bb9995916f1411bfb7b30aeeed20760e8d234fc3081c63cdaf09c WatchSource:0}: Error finding container 63013c3bc74bb9995916f1411bfb7b30aeeed20760e8d234fc3081c63cdaf09c: Status 404 returned error can't find the container with id 63013c3bc74bb9995916f1411bfb7b30aeeed20760e8d234fc3081c63cdaf09c Jul 06 19:26:00.529122 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:00.529100 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623f264d6030616a9f10578731099164.slice/crio-e2abb7d14440eb79d0edee15bbcd4f6b3599eae0786b3dc1f89f74285d7cf1e1 WatchSource:0}: Error finding container e2abb7d14440eb79d0edee15bbcd4f6b3599eae0786b3dc1f89f74285d7cf1e1: Status 404 returned error can't find the container with id e2abb7d14440eb79d0edee15bbcd4f6b3599eae0786b3dc1f89f74285d7cf1e1 Jul 06 19:26:00.533844 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.533830 2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:26:00.536608 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.536573 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:21:00 +0000 UTC" deadline="2028-02-18 23:59:52.483853461 +0000 UTC" Jul 06 19:26:00.536695 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.536609 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14212h33m51.947248918s" Jul 06 19:26:00.538725 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.538712 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:26:00.539810 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.539792 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:26:00.539885 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.539819 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:26:00.539885 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.539832 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:26:00.584667 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:00.584642 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Jul 06 19:26:00.600757 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.600717 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"623f264d6030616a9f10578731099164","Type":"ContainerStarted","Data":"e2abb7d14440eb79d0edee15bbcd4f6b3599eae0786b3dc1f89f74285d7cf1e1"} Jul 06 19:26:00.601700 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.601680 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" event={"ID":"ceedd80492377588ff46c4ba5374d82e","Type":"ContainerStarted","Data":"63013c3bc74bb9995916f1411bfb7b30aeeed20760e8d234fc3081c63cdaf09c"} Jul 06 19:26:00.608988 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.608970 2545 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:00.643056 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.643035 2545 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:00.663174 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.663151 2545 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:00.696940 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.696916 2545 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Jul 06 19:26:00.707516 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.707497 2545 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:26:00.708469 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.708457 2545 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Jul 06 19:26:00.721938 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:00.721922 2545 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:26:01.333905 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.333872 2545 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:26:01.475188 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.475159 2545 apiserver.go:52] "Watching apiserver" Jul 06 19:26:01.482169 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.482144 2545 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:26:01.482761 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.482726 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal","openshift-cluster-node-tuning-operator/tuned-cf56l","openshift-dns/node-resolver-tcfl6","openshift-image-registry/node-ca-lz9j9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal","openshift-multus/multus-4s74f","openshift-multus/network-metrics-daemon-sv6xg","openshift-network-diagnostics/network-check-target-m64lw","kube-system/global-pull-secret-syncer-cs4pr","kube-system/konnectivity-agent-9xtwb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g","openshift-multus/multus-additional-cni-plugins-m7v7k","openshift-network-operator/iptables-alerter-lv9fk","openshift-ovn-kubernetes/ovnkube-node-7mcmq"] Jul 06 19:26:01.484457 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.484415 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:01.484579 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.484530 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:01.485676 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.485647 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.487299 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.487798 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.487839 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.487899 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.487936 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fshrh\"" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.490570 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.490713 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.490793 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h2sqw\"" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.490925 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.491011 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:26:01.493475 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.492904 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.494160 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.494101 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:26:01.494956 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.494339 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:26:01.494956 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.494368 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:26:01.494956 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.494705 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:26:01.494956 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.494714 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jwdjk\"" Jul 06 19:26:01.495407 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.495388 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:26:01.495543 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.495524 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:26:01.495594 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.495564 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:26:01.495672 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.495656 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qtdsm\"" Jul 06 19:26:01.495755 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.495741 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:26:01.495793 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.495784 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:26:01.499323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.496330 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:26:01.499323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.496451 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:26:01.499323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.496664 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.499323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.496756 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jt8s4\"" Jul 06 19:26:01.499323 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.496760 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:01.499323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.496859 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:26:01.499323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.497211 2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:26:01.499323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.497665 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:26:01.500824 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.500360 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.502551 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.502530 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:26:01.502975 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.502959 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:26:01.503192 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.503176 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:26:01.503410 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.503394 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k5wcw\"" Jul 06 19:26:01.504053 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.504034 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.505560 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.505539 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:01.506323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.506303 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:26:01.506499 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.506483 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:26:01.506628 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.506609 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bjtmx\"" Jul 06 19:26:01.507659 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.507631 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:26:01.507862 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.507834 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:26:01.508041 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.508026 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gb97f\"" Jul 06 19:26:01.509077 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509057 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:01.509163 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.509143 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509266 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd9923b-0994-4f34-a64b-fb5de561aefd-hosts-file\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509302 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smjk\" (UniqueName: \"kubernetes.io/projected/0dd9923b-0994-4f34-a64b-fb5de561aefd-kube-api-access-9smjk\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509328 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-system-cni-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509361 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/662ad652-6785-41b0-843f-0968ee2564c8-iptables-alerter-script\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509386 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-multus-certs\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509419 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509444 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-registration-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509467 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-kubelet\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509533 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509559 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrbg4\" (UniqueName: \"kubernetes.io/projected/0b4dc7c3-5da3-4400-9da7-a00808f289b2-kube-api-access-lrbg4\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509584 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-k8s-cni-cncf-io\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509608 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509631 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-slash\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509684 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-run-netns\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.509742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509717 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-ovn\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509774 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-cni-bin\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509799 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-cni-multus\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509826 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-conf-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509850 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-systemd-units\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509878 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rzq\" (UniqueName: \"kubernetes.io/projected/662ad652-6785-41b0-843f-0968ee2564c8-kube-api-access-f6rzq\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.509991 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd9923b-0994-4f34-a64b-fb5de561aefd-tmp-dir\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510015 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-log-socket\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510056 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-socket-dir-parent\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510083 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/662ad652-6785-41b0-843f-0968ee2564c8-host-slash\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510120 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdd867de-45fa-4f74-9908-1ff7cb1314e9-dbus\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510153 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510179 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-etc-kubernetes\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510213 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a211f40a-7213-4ea6-9716-7b8e1a139c9e-serviceca\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510244 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-node-log\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510287 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-cni-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510316 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c77c4f21-bb4f-498b-ba80-631b38760d06-cni-binary-copy\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.510517 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510361 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdd867de-45fa-4f74-9908-1ff7cb1314e9-kubelet-config\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510401 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510449 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovn-node-metrics-cert\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510500 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-cnibin\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510554 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-netns\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510581 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510582 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-systemd\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510689 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.510965 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-sys-fs\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511012 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-cni-netd\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511041 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-os-release\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511093 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-cni-bin\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511122 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-kubelet\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511168 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-var-lib-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511200 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511226 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a211f40a-7213-4ea6-9716-7b8e1a139c9e-host\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.511328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511272 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9brl\" (UniqueName: \"kubernetes.io/projected/a211f40a-7213-4ea6-9716-7b8e1a139c9e-kube-api-access-c9brl\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511342 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovnkube-script-lib\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511374 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25vm2\" (UniqueName: \"kubernetes.io/projected/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-kube-api-access-25vm2\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511390 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-socket-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511440 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-device-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511466 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovnkube-config\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511513 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-env-overrides\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511551 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-hostroot\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511589 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshvv\" (UniqueName: \"kubernetes.io/projected/c77c4f21-bb4f-498b-ba80-631b38760d06-kube-api-access-pshvv\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511625 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjwq\" (UniqueName: \"kubernetes.io/projected/193f80e6-d6e0-4da5-8579-fb90a476df14-kube-api-access-ggjwq\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511648 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-daemon-config\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.512144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.511665 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-etc-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.512886 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.512868 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:26:01.513268 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.513247 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:26:01.513489 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.513251 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j6q6c\"" Jul 06 19:26:01.516133 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.516105 2545 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:26:01.546041 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.546020 2545 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7wzmt" Jul 06 19:26:01.559680 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.559651 2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7wzmt" Jul 06 19:26:01.598979 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.598907 2545 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:26:01.612405 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612366 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-systemd-units\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.612405 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612405 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rzq\" (UniqueName: \"kubernetes.io/projected/662ad652-6785-41b0-843f-0968ee2564c8-kube-api-access-f6rzq\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.612580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612454 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-lib-modules\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.612580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612481 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd9923b-0994-4f34-a64b-fb5de561aefd-tmp-dir\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.612580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612496 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-systemd-units\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.612580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612510 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-log-socket\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.612580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612553 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-cnibin\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.612580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612557 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-log-socket\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612586 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-socket-dir-parent\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612613 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/662ad652-6785-41b0-843f-0968ee2564c8-host-slash\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612637 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysctl-d\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612639 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-socket-dir-parent\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612660 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-sys\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612680 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/662ad652-6785-41b0-843f-0968ee2564c8-host-slash\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612688 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612724 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612752 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdd867de-45fa-4f74-9908-1ff7cb1314e9-dbus\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612757 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd9923b-0994-4f34-a64b-fb5de561aefd-tmp-dir\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612783 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:01.612855 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.612857 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612924 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-etc-kubernetes\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.612956 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:02.112929449 +0000 UTC m=+2.988476111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.612986 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-etc-kubernetes\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613011 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdd867de-45fa-4f74-9908-1ff7cb1314e9-dbus\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613026 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a211f40a-7213-4ea6-9716-7b8e1a139c9e-serviceca\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613059 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-node-log\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613080 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-systemd\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613181 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-cni-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613211 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c77c4f21-bb4f-498b-ba80-631b38760d06-cni-binary-copy\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.613414 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613231 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdd867de-45fa-4f74-9908-1ff7cb1314e9-kubelet-config\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613587 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a211f40a-7213-4ea6-9716-7b8e1a139c9e-serviceca\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613647 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613675 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovn-node-metrics-cert\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613703 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysctl-conf\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613704 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c77c4f21-bb4f-498b-ba80-631b38760d06-cni-binary-copy\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613736 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-node-log\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613772 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-host\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613789 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f53f1f29-1c7d-473f-b148-fab71b101e48-konnectivity-ca\") pod \"konnectivity-agent-9xtwb\" (UID: \"f53f1f29-1c7d-473f-b148-fab71b101e48\") " pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613819 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-cnibin\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613844 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-netns\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613860 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-systemd\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.613926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613908 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613930 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-os-release\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613963 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-sys-fs\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.613987 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-cni-netd\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614011 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysconfig\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614028 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-system-cni-dir\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614043 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-os-release\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614058 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-cni-bin\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614080 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-kubelet\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614132 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-var-lib-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614134 2545 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614149 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-cni-netd\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614157 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-modprobe-d\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614193 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-kubelet\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614199 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d349367-e802-4593-aa7b-3702c64f7c5d-tmp\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614226 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-systemd\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614243 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614272 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-cni-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.614468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614273 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a211f40a-7213-4ea6-9716-7b8e1a139c9e-host\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614301 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a211f40a-7213-4ea6-9716-7b8e1a139c9e-host\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614309 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9brl\" (UniqueName: \"kubernetes.io/projected/a211f40a-7213-4ea6-9716-7b8e1a139c9e-kube-api-access-c9brl\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614338 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovnkube-script-lib\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614401 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614461 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-tuned\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614492 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgwnm\" (UniqueName: \"kubernetes.io/projected/8d349367-e802-4593-aa7b-3702c64f7c5d-kube-api-access-bgwnm\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614524 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25vm2\" (UniqueName: \"kubernetes.io/projected/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-kube-api-access-25vm2\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614551 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-socket-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614577 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-cni-bin\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614577 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-device-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614612 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614630 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovnkube-config\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614656 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-env-overrides\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614673 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-sys-fs\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614676 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-run\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614708 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-var-lib-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.615369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614714 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-var-lib-kubelet\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614952 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-socket-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614957 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f53f1f29-1c7d-473f-b148-fab71b101e48-agent-certs\") pod \"konnectivity-agent-9xtwb\" (UID: \"f53f1f29-1c7d-473f-b148-fab71b101e48\") " pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614994 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-hostroot\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.614999 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-device-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615019 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshvv\" (UniqueName: \"kubernetes.io/projected/c77c4f21-bb4f-498b-ba80-631b38760d06-kube-api-access-pshvv\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615036 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjwq\" (UniqueName: \"kubernetes.io/projected/193f80e6-d6e0-4da5-8579-fb90a476df14-kube-api-access-ggjwq\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615057 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5m8\" (UniqueName: \"kubernetes.io/projected/14d0d201-5971-445b-80f5-c4addff50780-kube-api-access-hn5m8\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615082 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-daemon-config\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615112 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovnkube-script-lib\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615148 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-etc-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615169 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615182 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-cnibin\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615197 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd9923b-0994-4f34-a64b-fb5de561aefd-hosts-file\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615214 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smjk\" (UniqueName: \"kubernetes.io/projected/0dd9923b-0994-4f34-a64b-fb5de561aefd-kube-api-access-9smjk\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615220 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-netns\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615250 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-system-cni-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.616232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615277 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-os-release\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615281 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/662ad652-6785-41b0-843f-0968ee2564c8-iptables-alerter-script\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615314 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-multus-certs\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615387 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-hostroot\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615387 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovnkube-config\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615422 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615663 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615696 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-registration-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615726 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-kubelet\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615780 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-daemon-config\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615840 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615783 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615881 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrbg4\" (UniqueName: \"kubernetes.io/projected/0b4dc7c3-5da3-4400-9da7-a00808f289b2-kube-api-access-lrbg4\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615911 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-kubernetes\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615935 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615945 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-multus-certs\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.615781 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b4dc7c3-5da3-4400-9da7-a00808f289b2-env-overrides\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616001 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-k8s-cni-cncf-io\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616032 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616033 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-etc-openvswitch\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616060 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-slash\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616089 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-run-netns\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.616108 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616115 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-ovn\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616146 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-cni-bin\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.616156 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret podName:fdd867de-45fa-4f74-9908-1ff7cb1314e9 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:02.116144204 +0000 UTC m=+2.991690865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret") pod "global-pull-secret-syncer-cs4pr" (UID: "fdd867de-45fa-4f74-9908-1ff7cb1314e9") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616211 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-cni-bin\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616222 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-registration-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616247 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-cni-binary-copy\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616268 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/193f80e6-d6e0-4da5-8579-fb90a476df14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616271 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-cni-multus\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616290 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-conf-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616307 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-system-cni-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616356 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-multus-conf-dir\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.617683 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616395 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd9923b-0994-4f34-a64b-fb5de561aefd-hosts-file\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.618298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616391 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-slash\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.618298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616448 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-run-netns\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.618298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616448 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdd867de-45fa-4f74-9908-1ff7cb1314e9-kubelet-config\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:01.618298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616515 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-run-k8s-cni-cncf-io\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.618298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616550 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/662ad652-6785-41b0-843f-0968ee2564c8-iptables-alerter-script\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.618298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616554 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-run-ovn\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.618298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616624 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b4dc7c3-5da3-4400-9da7-a00808f289b2-host-kubelet\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.618298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.616669 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c77c4f21-bb4f-498b-ba80-631b38760d06-host-var-lib-cni-multus\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.620938 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.620916 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b4dc7c3-5da3-4400-9da7-a00808f289b2-ovn-node-metrics-cert\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.625845 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.625819 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rzq\" (UniqueName: \"kubernetes.io/projected/662ad652-6785-41b0-843f-0968ee2564c8-kube-api-access-f6rzq\") pod \"iptables-alerter-lv9fk\" (UID: \"662ad652-6785-41b0-843f-0968ee2564c8\") " pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.626099 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.626080 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25vm2\" (UniqueName: \"kubernetes.io/projected/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-kube-api-access-25vm2\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:01.626183 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.626161 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9brl\" (UniqueName: \"kubernetes.io/projected/a211f40a-7213-4ea6-9716-7b8e1a139c9e-kube-api-access-c9brl\") pod \"node-ca-lz9j9\" (UID: \"a211f40a-7213-4ea6-9716-7b8e1a139c9e\") " pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.626697 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.626674 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjwq\" (UniqueName: \"kubernetes.io/projected/193f80e6-d6e0-4da5-8579-fb90a476df14-kube-api-access-ggjwq\") pod \"aws-ebs-csi-driver-node-nnn9g\" (UID: \"193f80e6-d6e0-4da5-8579-fb90a476df14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.627263 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.627243 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshvv\" (UniqueName: \"kubernetes.io/projected/c77c4f21-bb4f-498b-ba80-631b38760d06-kube-api-access-pshvv\") pod \"multus-4s74f\" (UID: \"c77c4f21-bb4f-498b-ba80-631b38760d06\") " pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.628320 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.628298 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrbg4\" (UniqueName: \"kubernetes.io/projected/0b4dc7c3-5da3-4400-9da7-a00808f289b2-kube-api-access-lrbg4\") pod \"ovnkube-node-7mcmq\" (UID: \"0b4dc7c3-5da3-4400-9da7-a00808f289b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.630681 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.630660 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smjk\" (UniqueName: \"kubernetes.io/projected/0dd9923b-0994-4f34-a64b-fb5de561aefd-kube-api-access-9smjk\") pod \"node-resolver-tcfl6\" (UID: \"0dd9923b-0994-4f34-a64b-fb5de561aefd\") " pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.717292 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717262 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-kubernetes\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.717292 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717295 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717331 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-cni-binary-copy\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717359 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-lib-modules\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717384 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-cnibin\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717395 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-kubernetes\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717411 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysctl-d\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717438 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-sys\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717462 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717463 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-cnibin\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717522 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717528 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-lib-modules\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717551 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717532 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-sys\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.717572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717557 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysctl-d\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717581 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-systemd\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717610 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysctl-conf\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717626 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-host\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717632 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-systemd\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717642 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f53f1f29-1c7d-473f-b148-fab71b101e48-konnectivity-ca\") pod \"konnectivity-agent-9xtwb\" (UID: \"f53f1f29-1c7d-473f-b148-fab71b101e48\") " pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717664 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-os-release\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717690 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysconfig\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717705 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-host\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717715 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-system-cni-dir\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717746 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-modprobe-d\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717761 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysctl-conf\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717770 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-os-release\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717771 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d349367-e802-4593-aa7b-3702c64f7c5d-tmp\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717775 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-sysconfig\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717818 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14d0d201-5971-445b-80f5-c4addff50780-system-cni-dir\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717820 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-tuned\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717861 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgwnm\" (UniqueName: \"kubernetes.io/projected/8d349367-e802-4593-aa7b-3702c64f7c5d-kube-api-access-bgwnm\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717894 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-run\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717897 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-modprobe-d\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717921 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-var-lib-kubelet\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717945 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f53f1f29-1c7d-473f-b148-fab71b101e48-agent-certs\") pod \"konnectivity-agent-9xtwb\" (UID: \"f53f1f29-1c7d-473f-b148-fab71b101e48\") " pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.717980 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5m8\" (UniqueName: \"kubernetes.io/projected/14d0d201-5971-445b-80f5-c4addff50780-kube-api-access-hn5m8\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.718008 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.718051 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-cni-binary-copy\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.718076 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.718114 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-var-lib-kubelet\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.718155 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d349367-e802-4593-aa7b-3702c64f7c5d-run\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.718336 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f53f1f29-1c7d-473f-b148-fab71b101e48-konnectivity-ca\") pod \"konnectivity-agent-9xtwb\" (UID: \"f53f1f29-1c7d-473f-b148-fab71b101e48\") " pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:01.718920 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.718588 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14d0d201-5971-445b-80f5-c4addff50780-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.720187 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.720162 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d349367-e802-4593-aa7b-3702c64f7c5d-etc-tuned\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.720320 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.720301 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d349367-e802-4593-aa7b-3702c64f7c5d-tmp\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.720700 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.720646 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f53f1f29-1c7d-473f-b148-fab71b101e48-agent-certs\") pod \"konnectivity-agent-9xtwb\" (UID: \"f53f1f29-1c7d-473f-b148-fab71b101e48\") " pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:01.728460 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.728438 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:01.728460 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.728464 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:01.728601 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.728478 2545 projected.go:196] Error preparing data for projected volume kube-api-access-gklk4 for pod openshift-network-diagnostics/network-check-target-m64lw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:01.728601 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:01.728545 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4 podName:49f0b1f7-bd6e-4377-b5ca-896c1d077438 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:02.228525546 +0000 UTC m=+3.104072212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gklk4" (UniqueName: "kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4") pod "network-check-target-m64lw" (UID: "49f0b1f7-bd6e-4377-b5ca-896c1d077438") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:01.735657 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.735616 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5m8\" (UniqueName: \"kubernetes.io/projected/14d0d201-5971-445b-80f5-c4addff50780-kube-api-access-hn5m8\") pod \"multus-additional-cni-plugins-m7v7k\" (UID: \"14d0d201-5971-445b-80f5-c4addff50780\") " pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:01.735769 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.735738 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgwnm\" (UniqueName: \"kubernetes.io/projected/8d349367-e802-4593-aa7b-3702c64f7c5d-kube-api-access-bgwnm\") pod \"tuned-cf56l\" (UID: \"8d349367-e802-4593-aa7b-3702c64f7c5d\") " pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.804572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.804542 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" Jul 06 19:26:01.812762 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.812739 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tcfl6" Jul 06 19:26:01.822263 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.822238 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lz9j9" Jul 06 19:26:01.828747 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.828728 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:01.837242 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.837216 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4s74f" Jul 06 19:26:01.845185 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.845163 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lv9fk" Jul 06 19:26:01.853898 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.853830 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cf56l" Jul 06 19:26:01.862656 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.862638 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:01.868515 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:01.868489 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" Jul 06 19:26:02.121874 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:02.121795 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:02.122029 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:02.121888 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:02.122029 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.121924 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:02.122029 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.121985 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:02.122029 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.122003 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:03.121981674 +0000 UTC m=+3.997528336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:02.122029 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.122026 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret podName:fdd867de-45fa-4f74-9908-1ff7cb1314e9 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:03.122013303 +0000 UTC m=+3.997559963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret") pod "global-pull-secret-syncer-cs4pr" (UID: "fdd867de-45fa-4f74-9908-1ff7cb1314e9") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:02.323741 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:02.323710 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:02.323898 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.323845 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:02.323898 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.323859 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:02.323898 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.323869 2545 projected.go:196] Error preparing data for projected volume kube-api-access-gklk4 for pod openshift-network-diagnostics/network-check-target-m64lw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:02.323999 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.323922 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4 podName:49f0b1f7-bd6e-4377-b5ca-896c1d077438 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:03.323903984 +0000 UTC m=+4.199450665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gklk4" (UniqueName: "kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4") pod "network-check-target-m64lw" (UID: "49f0b1f7-bd6e-4377-b5ca-896c1d077438") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:02.562662 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:02.562594 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:21:02 +0000 UTC" deadline="2028-03-27 05:28:07.55049388 +0000 UTC" Jul 06 19:26:02.562662 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:02.562650 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15106h2m4.987848794s" Jul 06 19:26:02.598976 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:02.598953 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:02.599094 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:02.599053 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:02.625891 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:02.625853 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193f80e6_d6e0_4da5_8579_fb90a476df14.slice/crio-c573467ad5de57a3e054122a460a36c0f2f94e16afde0ca75e9a3a87eea9ce65 WatchSource:0}: Error finding container c573467ad5de57a3e054122a460a36c0f2f94e16afde0ca75e9a3a87eea9ce65: Status 404 returned error can't find the container with id c573467ad5de57a3e054122a460a36c0f2f94e16afde0ca75e9a3a87eea9ce65 Jul 06 19:26:02.633153 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:02.633017 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662ad652_6785_41b0_843f_0968ee2564c8.slice/crio-02de4accd4a271da453826150af6d0e5c9f5ea60cd35bde289226704dd515873 WatchSource:0}: Error finding container 02de4accd4a271da453826150af6d0e5c9f5ea60cd35bde289226704dd515873: Status 404 returned error can't find the container with id 02de4accd4a271da453826150af6d0e5c9f5ea60cd35bde289226704dd515873 Jul 06 19:26:02.634062 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:02.634037 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda211f40a_7213_4ea6_9716_7b8e1a139c9e.slice/crio-03819522c77acc80a04ee14fd69e628a0f858ebb447783cbc7be341fab9a297e WatchSource:0}: Error finding container 03819522c77acc80a04ee14fd69e628a0f858ebb447783cbc7be341fab9a297e: Status 404 returned error can't find the container with id 03819522c77acc80a04ee14fd69e628a0f858ebb447783cbc7be341fab9a297e Jul 06 19:26:02.635877 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:02.635853 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d0d201_5971_445b_80f5_c4addff50780.slice/crio-a1050db811718614ea5ff9255b02aff0d439b587dcfb9e35c50fe3eb80227139 WatchSource:0}: Error finding container a1050db811718614ea5ff9255b02aff0d439b587dcfb9e35c50fe3eb80227139: Status 404 returned error can't find the container with id a1050db811718614ea5ff9255b02aff0d439b587dcfb9e35c50fe3eb80227139 Jul 06 19:26:02.637533 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:02.637515 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d349367_e802_4593_aa7b_3702c64f7c5d.slice/crio-47a54e4d14f47d5f9bda69fbcd0643e65f1c33acaee9901038394892c14421f7 WatchSource:0}: Error finding container 47a54e4d14f47d5f9bda69fbcd0643e65f1c33acaee9901038394892c14421f7: Status 404 returned error can't find the container with id 47a54e4d14f47d5f9bda69fbcd0643e65f1c33acaee9901038394892c14421f7 Jul 06 19:26:02.638073 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:02.637990 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53f1f29_1c7d_473f_b148_fab71b101e48.slice/crio-296796fafddc950bc55afc86a62b26f6b1264715caf2ceff1dbe8ba8e2244229 WatchSource:0}: Error finding container 296796fafddc950bc55afc86a62b26f6b1264715caf2ceff1dbe8ba8e2244229: Status 404 returned error can't find the container with id 296796fafddc950bc55afc86a62b26f6b1264715caf2ceff1dbe8ba8e2244229 Jul 06 19:26:02.639999 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:02.639967 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4dc7c3_5da3_4400_9da7_a00808f289b2.slice/crio-a9b14807affca234b0e21e35e5eb617412bbf9a58cbb09957738c8b360403138 WatchSource:0}: Error finding container a9b14807affca234b0e21e35e5eb617412bbf9a58cbb09957738c8b360403138: Status 404 returned error can't find the container with id a9b14807affca234b0e21e35e5eb617412bbf9a58cbb09957738c8b360403138 Jul 06 19:26:03.129415 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.128615 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:03.129415 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.128695 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:03.129415 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.128856 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:03.129415 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.128922 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:05.12890147 +0000 UTC m=+6.004448134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:03.129415 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.129328 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:03.129415 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.129378 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret podName:fdd867de-45fa-4f74-9908-1ff7cb1314e9 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:05.129362852 +0000 UTC m=+6.004909530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret") pod "global-pull-secret-syncer-cs4pr" (UID: "fdd867de-45fa-4f74-9908-1ff7cb1314e9") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:03.330961 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.330230 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:03.330961 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.330431 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:03.330961 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.330473 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:03.330961 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.330487 2545 projected.go:196] Error preparing data for projected volume kube-api-access-gklk4 for pod openshift-network-diagnostics/network-check-target-m64lw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:03.330961 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.330552 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4 podName:49f0b1f7-bd6e-4377-b5ca-896c1d077438 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:05.330529773 +0000 UTC m=+6.206076439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gklk4" (UniqueName: "kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4") pod "network-check-target-m64lw" (UID: "49f0b1f7-bd6e-4377-b5ca-896c1d077438") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:03.563579 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.563367 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:21:02 +0000 UTC" deadline="2028-01-05 20:42:02.704996908 +0000 UTC" Jul 06 19:26:03.563579 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.563419 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13153h15m59.141583161s" Jul 06 19:26:03.598852 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.598756 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:03.599024 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.598870 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:03.599332 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.599250 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:03.599409 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:03.599352 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:03.615860 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.615715 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"a9b14807affca234b0e21e35e5eb617412bbf9a58cbb09957738c8b360403138"} Jul 06 19:26:03.635012 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.634975 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lz9j9" event={"ID":"a211f40a-7213-4ea6-9716-7b8e1a139c9e","Type":"ContainerStarted","Data":"03819522c77acc80a04ee14fd69e628a0f858ebb447783cbc7be341fab9a297e"} Jul 06 19:26:03.651898 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.651858 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9xtwb" event={"ID":"f53f1f29-1c7d-473f-b148-fab71b101e48","Type":"ContainerStarted","Data":"296796fafddc950bc55afc86a62b26f6b1264715caf2ceff1dbe8ba8e2244229"} Jul 06 19:26:03.663655 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.663621 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cf56l" event={"ID":"8d349367-e802-4593-aa7b-3702c64f7c5d","Type":"ContainerStarted","Data":"47a54e4d14f47d5f9bda69fbcd0643e65f1c33acaee9901038394892c14421f7"} Jul 06 19:26:03.670168 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.670139 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lv9fk" event={"ID":"662ad652-6785-41b0-843f-0968ee2564c8","Type":"ContainerStarted","Data":"02de4accd4a271da453826150af6d0e5c9f5ea60cd35bde289226704dd515873"} Jul 06 19:26:03.686817 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.686725 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4s74f" event={"ID":"c77c4f21-bb4f-498b-ba80-631b38760d06","Type":"ContainerStarted","Data":"12359f5c767bd5eaf13ae0c6e0b03c198476add19bf98084deb0d54cd8ab16dd"} Jul 06 19:26:03.692019 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.691831 2545 generic.go:356] "Generic (PLEG): container finished" podID="623f264d6030616a9f10578731099164" containerID="a40df6471522ad4e076269145ad52925fb24e28db65bb2938b2ea8af71f0f4c1" exitCode=0 Jul 06 19:26:03.692019 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.691907 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"623f264d6030616a9f10578731099164","Type":"ContainerDied","Data":"a40df6471522ad4e076269145ad52925fb24e28db65bb2938b2ea8af71f0f4c1"} Jul 06 19:26:03.721684 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.717240 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerStarted","Data":"a1050db811718614ea5ff9255b02aff0d439b587dcfb9e35c50fe3eb80227139"} Jul 06 19:26:03.746538 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.746463 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tcfl6" event={"ID":"0dd9923b-0994-4f34-a64b-fb5de561aefd","Type":"ContainerStarted","Data":"29f3927742cc2c7cf964215b75465e09b64d84ebe22037f99fc4198219d3895f"} Jul 06 19:26:03.753862 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.753829 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" event={"ID":"193f80e6-d6e0-4da5-8579-fb90a476df14","Type":"ContainerStarted","Data":"c573467ad5de57a3e054122a460a36c0f2f94e16afde0ca75e9a3a87eea9ce65"} Jul 06 19:26:03.770604 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:03.770550 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" event={"ID":"ceedd80492377588ff46c4ba5374d82e","Type":"ContainerStarted","Data":"d6b5a05c47aca48806b84d32a49c17082f87cf9bdd0d5b1de19183e719e429ca"} Jul 06 19:26:04.598853 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:04.598816 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:04.599329 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:04.598962 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:04.787699 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:04.787560 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"623f264d6030616a9f10578731099164","Type":"ContainerStarted","Data":"a151c960077a4018214c306dc157053d8a4d132282545d688b685f5d75b1b2e9"} Jul 06 19:26:04.807652 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:04.807008 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" podStartSLOduration=3.806989737 podStartE2EDuration="3.806989737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:03.801524917 +0000 UTC m=+4.677071602" watchObservedRunningTime="2026-07-06 19:26:04.806989737 +0000 UTC m=+5.682536415" Jul 06 19:26:05.149312 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:05.148509 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:05.149312 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:05.148583 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:05.149312 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.148746 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:05.149312 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.148810 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:09.148790186 +0000 UTC m=+10.024336851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:05.149312 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.149220 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:05.149312 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.149270 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret podName:fdd867de-45fa-4f74-9908-1ff7cb1314e9 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:09.149254243 +0000 UTC m=+10.024800931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret") pod "global-pull-secret-syncer-cs4pr" (UID: "fdd867de-45fa-4f74-9908-1ff7cb1314e9") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:05.350903 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:05.350861 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:05.351070 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.351054 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:05.351146 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.351076 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:05.351146 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.351090 2545 projected.go:196] Error preparing data for projected volume kube-api-access-gklk4 for pod openshift-network-diagnostics/network-check-target-m64lw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:05.351249 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.351153 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4 podName:49f0b1f7-bd6e-4377-b5ca-896c1d077438 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:09.351133264 +0000 UTC m=+10.226679944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gklk4" (UniqueName: "kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4") pod "network-check-target-m64lw" (UID: "49f0b1f7-bd6e-4377-b5ca-896c1d077438") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:05.600838 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:05.600741 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:05.601370 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.600863 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:05.601370 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:05.600934 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:05.601370 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:05.601017 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:06.598996 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:06.598951 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:06.599210 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:06.599092 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:07.599635 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:07.599595 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:07.600107 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:07.599786 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:07.600404 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:07.600233 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:07.600404 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:07.600337 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:08.599097 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:08.599063 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:08.599288 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:08.599206 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:09.187419 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:09.187378 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:09.187978 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:09.187482 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:09.187978 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.187643 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:09.187978 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.187693 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:17.187679561 +0000 UTC m=+18.063226222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:09.188218 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.188049 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:09.188218 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.188093 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret podName:fdd867de-45fa-4f74-9908-1ff7cb1314e9 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:17.188082139 +0000 UTC m=+18.063628801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret") pod "global-pull-secret-syncer-cs4pr" (UID: "fdd867de-45fa-4f74-9908-1ff7cb1314e9") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:09.390188 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:09.389576 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:09.390188 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.389754 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:09.390188 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.389779 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:09.390188 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.389792 2545 projected.go:196] Error preparing data for projected volume kube-api-access-gklk4 for pod openshift-network-diagnostics/network-check-target-m64lw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:09.390188 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.389853 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4 podName:49f0b1f7-bd6e-4377-b5ca-896c1d077438 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:17.389835375 +0000 UTC m=+18.265382041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gklk4" (UniqueName: "kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4") pod "network-check-target-m64lw" (UID: "49f0b1f7-bd6e-4377-b5ca-896c1d077438") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:09.600968 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:09.599855 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:09.600968 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.599967 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:09.600968 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:09.600445 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:09.600968 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:09.600541 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:10.598673 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:10.598638 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:10.599134 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:10.598769 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:11.600308 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:11.598557 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:11.600308 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:11.598564 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:11.600308 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:11.598712 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:11.600308 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:11.598785 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:12.599238 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:12.598805 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:12.599238 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:12.598930 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:13.598604 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:13.598569 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:13.599032 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:13.598569 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:13.599032 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:13.598711 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:13.599032 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:13.598737 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:14.598019 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:14.597954 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:14.598146 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:14.598078 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:15.599014 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:15.598974 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:15.599408 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:15.599110 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:15.599499 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:15.599434 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:15.599546 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:15.599521 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:16.598916 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:16.598883 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:16.599126 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:16.598992 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:17.246447 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:17.246407 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:17.246626 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:17.246482 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:17.246626 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.246587 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:17.246626 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.246587 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:17.246750 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.246650 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:33.246630475 +0000 UTC m=+34.122177156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:17.246750 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.246664 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret podName:fdd867de-45fa-4f74-9908-1ff7cb1314e9 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:33.246657802 +0000 UTC m=+34.122204461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret") pod "global-pull-secret-syncer-cs4pr" (UID: "fdd867de-45fa-4f74-9908-1ff7cb1314e9") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:17.447980 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:17.447942 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:17.448162 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.448072 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:17.448162 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.448090 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:17.448162 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.448100 2545 projected.go:196] Error preparing data for projected volume kube-api-access-gklk4 for pod openshift-network-diagnostics/network-check-target-m64lw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:17.448162 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.448159 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4 podName:49f0b1f7-bd6e-4377-b5ca-896c1d077438 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:33.44814207 +0000 UTC m=+34.323688754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gklk4" (UniqueName: "kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4") pod "network-check-target-m64lw" (UID: "49f0b1f7-bd6e-4377-b5ca-896c1d077438") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:17.599321 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:17.598770 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:17.599321 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.598920 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:17.599321 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:17.599017 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:17.599321 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:17.599136 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:18.598931 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:18.598900 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:18.599122 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:18.599022 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:19.598765 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:19.598734 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:19.599201 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:19.598849 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:19.599201 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:19.598938 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:19.599201 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:19.599054 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:20.598316 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:20.598283 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:20.598532 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:20.598440 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:21.598315 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:21.598285 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:21.598894 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:21.598440 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:21.598894 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:21.598483 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:21.598894 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:21.598597 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:22.598474 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.598171 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:22.599088 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:22.598558 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:22.829826 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.829802 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:26:22.830139 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.830113 2545 generic.go:356] "Generic (PLEG): container finished" podID="0b4dc7c3-5da3-4400-9da7-a00808f289b2" containerID="d9701c172fe0cddeaa56c62c93473d9f3a6ecb48aa5819963a86a657a0f9b0f1" exitCode=1 Jul 06 19:26:22.830231 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.830185 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"a1bbfdb4314a2a0c8ab43db4f1740d50094b77b021b8960c41af4d2cc7be205f"} Jul 06 19:26:22.830303 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.830288 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"f225de5c0ef293c9bc48ab9a593377bea051384d9d1f38e15ddb8ca6df2384b9"} Jul 06 19:26:22.830356 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.830342 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"0f4d955f765b097ef733ed8ecc34f2cca07cbd66ed4684c7f939004de311059c"} Jul 06 19:26:22.830411 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.830394 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerDied","Data":"d9701c172fe0cddeaa56c62c93473d9f3a6ecb48aa5819963a86a657a0f9b0f1"} Jul 06 19:26:22.830471 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.830458 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"f356f0b3740fe70c871b9db4253a45a54afd2eeda0d3d795fbadf7f0cb2893ab"} Jul 06 19:26:22.831565 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.831547 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lz9j9" event={"ID":"a211f40a-7213-4ea6-9716-7b8e1a139c9e","Type":"ContainerStarted","Data":"7c3e6a1dee338c2afbad78d1b2d7db2475ebfad7b6aa1e747a77d0ccab246c08"} Jul 06 19:26:22.832896 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.832877 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9xtwb" event={"ID":"f53f1f29-1c7d-473f-b148-fab71b101e48","Type":"ContainerStarted","Data":"f51b70b332cf296a2c3635f5c2d0f8dbbda9d8e03e5ada008184fa3dc91b8ba6"} Jul 06 19:26:22.834214 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.834196 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cf56l" event={"ID":"8d349367-e802-4593-aa7b-3702c64f7c5d","Type":"ContainerStarted","Data":"e89d26d95a68e3b462605ae07c89698f4c138276955fa49215e8bf44088ad2db"} Jul 06 19:26:22.835502 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.835484 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4s74f" event={"ID":"c77c4f21-bb4f-498b-ba80-631b38760d06","Type":"ContainerStarted","Data":"ed2e2b886e39254af092c258e39edc1669a0b0982612ae31f0cfc6124b501a57"} Jul 06 19:26:22.836880 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.836860 2545 generic.go:356] "Generic (PLEG): container finished" podID="14d0d201-5971-445b-80f5-c4addff50780" containerID="7b69429e6d2c44b1ba89d3f390db760b980a49916ae6d1c887dd075933fec849" exitCode=0 Jul 06 19:26:22.836942 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.836929 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerDied","Data":"7b69429e6d2c44b1ba89d3f390db760b980a49916ae6d1c887dd075933fec849"} Jul 06 19:26:22.838310 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.838292 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tcfl6" event={"ID":"0dd9923b-0994-4f34-a64b-fb5de561aefd","Type":"ContainerStarted","Data":"cc7f4206c4d7882cb98378b68b9f8f87dad5a426682fbe9670a02c7634f385f2"} Jul 06 19:26:22.839743 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.839727 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" event={"ID":"193f80e6-d6e0-4da5-8579-fb90a476df14","Type":"ContainerStarted","Data":"bc5cba778fca33315c30ceee5a68f6fe6685e54d20c182e09a20c53faf5dee31"} Jul 06 19:26:22.846833 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.846787 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lz9j9" podStartSLOduration=3.337571302 podStartE2EDuration="22.846748154s" totalImagesPullingTime="19.509176852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:22.846667449 +0000 UTC m=+23.722214135" watchObservedRunningTime="2026-07-06 19:26:22.846748154 +0000 UTC m=+23.722294838" Jul 06 19:26:22.846970 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.846943 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" podStartSLOduration=21.846929859 podStartE2EDuration="21.846929859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:04.807621 +0000 UTC m=+5.683167684" watchObservedRunningTime="2026-07-06 19:26:22.846929859 +0000 UTC m=+23.722476540" Jul 06 19:26:22.863638 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.863576 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tcfl6" podStartSLOduration=3.341823105 podStartE2EDuration="22.863566623s" totalImagesPullingTime="19.521743518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:22.863245237 +0000 UTC m=+23.738791909" watchObservedRunningTime="2026-07-06 19:26:22.863566623 +0000 UTC m=+23.739113301" Jul 06 19:26:22.936628 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.936576 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4s74f" podStartSLOduration=3.360459439 podStartE2EDuration="22.936560364s" totalImagesPullingTime="19.576100925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:22.89844205 +0000 UTC m=+23.773988732" watchObservedRunningTime="2026-07-06 19:26:22.936560364 +0000 UTC m=+23.812107044" Jul 06 19:26:22.958175 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.958136 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cf56l" podStartSLOduration=3.433215309 podStartE2EDuration="22.95812585s" totalImagesPullingTime="19.524910541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:22.936690067 +0000 UTC m=+23.812236734" watchObservedRunningTime="2026-07-06 19:26:22.95812585 +0000 UTC m=+23.833672530" Jul 06 19:26:22.978984 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:22.978913 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9xtwb" podStartSLOduration=3.516222688 podStartE2EDuration="22.978894931s" totalImagesPullingTime="19.462672243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:22.957886769 +0000 UTC m=+23.833433450" watchObservedRunningTime="2026-07-06 19:26:22.978894931 +0000 UTC m=+23.854441612" Jul 06 19:26:23.598971 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:23.598916 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:23.599356 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:23.599039 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:23.599356 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:23.599077 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:23.599356 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:23.599171 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:23.805373 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:23.805350 2545 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:26:23.843687 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:23.843660 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" event={"ID":"193f80e6-d6e0-4da5-8579-fb90a476df14","Type":"ContainerStarted","Data":"b0ccd63af0040376cb5092d66bb8a0ba8b09a5f9ca68be0da0d2cecd316812f2"} Jul 06 19:26:23.846780 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:23.846763 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:26:23.847139 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:23.847118 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"05e07315e593cbaef1254e2f21f2e1afcd2f70a6054b41d50251b2d4d752dd6c"} Jul 06 19:26:23.848785 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:23.848759 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lv9fk" event={"ID":"662ad652-6785-41b0-843f-0968ee2564c8","Type":"ContainerStarted","Data":"7acc1140813135b0c5221b9846015afbb0fc751468d76099db00bcec7975fb31"} Jul 06 19:26:24.566448 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:24.565714 2545 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:26:23.80536817Z","UUID":"9f815451-a422-4831-8a92-7a2841a9bbdc","Handler":null,"Name":"","Endpoint":""} Jul 06 19:26:24.567460 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:24.567438 2545 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:26:24.567575 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:24.567484 2545 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:26:24.598345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:24.598320 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:24.598500 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:24.598432 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:25.598311 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:25.598282 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:25.598830 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:25.598435 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:25.598830 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:25.598464 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:25.598830 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:25.598566 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:25.855514 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:25.855240 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" event={"ID":"193f80e6-d6e0-4da5-8579-fb90a476df14","Type":"ContainerStarted","Data":"e0c96e4a0e4c0e6cfcfaec39c92aa5bb2119474b343b8644543e5a4b2c1150db"} Jul 06 19:26:25.857997 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:25.857978 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:26:25.858293 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:25.858273 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"aaffb0b40fc0f03b54aef034c0444080fe8ff78edec3e31c4b21c373dfbd9d7c"} Jul 06 19:26:25.874780 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:25.874742 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnn9g" podStartSLOduration=3.909034093 podStartE2EDuration="25.874730097s" totalImagesPullingTime="21.965696004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:25.874332317 +0000 UTC m=+26.749879000" watchObservedRunningTime="2026-07-06 19:26:25.874730097 +0000 UTC m=+26.750276777" Jul 06 19:26:25.874905 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:25.874885 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lv9fk" podStartSLOduration=6.358804484 podStartE2EDuration="25.874882287s" totalImagesPullingTime="19.516077803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:23.880640355 +0000 UTC m=+24.756187041" watchObservedRunningTime="2026-07-06 19:26:25.874882287 +0000 UTC m=+26.750428967" Jul 06 19:26:26.598736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:26.598698 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:26.599289 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:26.598819 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:26.661182 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:26.661153 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:26.661752 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:26.661732 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:26.860314 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:26.860204 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:26.860578 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:26.860560 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9xtwb" Jul 06 19:26:27.598762 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:27.598733 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:27.599323 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:27.598860 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:27.599323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:27.598900 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:27.599323 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:27.598963 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:28.598169 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:28.598030 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:28.598286 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:28.598257 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:28.868245 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:28.868184 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:26:28.868825 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:28.868556 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"ff1226fda9dcfb7420659efd872e1aeab396675dc1a14dea41715b0606eb7edb"} Jul 06 19:26:28.868928 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:28.868906 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:28.869053 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:28.868985 2545 scope.go:117] "RemoveContainer" containerID="d9701c172fe0cddeaa56c62c93473d9f3a6ecb48aa5819963a86a657a0f9b0f1" Jul 06 19:26:28.870239 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:28.870221 2545 generic.go:356] "Generic (PLEG): container finished" podID="14d0d201-5971-445b-80f5-c4addff50780" containerID="68e5a2e15e66d1924cb1eb0f58edd9c464a4cbcc1e96c05cc1e62cd0290c0fcd" exitCode=0 Jul 06 19:26:28.870343 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:28.870318 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerDied","Data":"68e5a2e15e66d1924cb1eb0f58edd9c464a4cbcc1e96c05cc1e62cd0290c0fcd"} Jul 06 19:26:28.885029 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:28.885013 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:29.598270 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:29.598245 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:29.598581 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:29.598356 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:29.599819 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:29.599795 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:29.599939 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:29.599916 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:29.876467 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:29.876390 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:26:29.876904 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:29.876836 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" event={"ID":"0b4dc7c3-5da3-4400-9da7-a00808f289b2","Type":"ContainerStarted","Data":"60d260e91b280a2e057af206be8703963fe268d873c687cdffa949cbd9e5f365"} Jul 06 19:26:29.877082 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:29.877060 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:29.877168 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:29.877142 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:29.892978 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:29.892956 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:26:29.911112 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:29.911050 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" podStartSLOduration=10.322833068 podStartE2EDuration="29.911029771s" totalImagesPullingTime="19.588196703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:29.910840982 +0000 UTC m=+30.786387676" watchObservedRunningTime="2026-07-06 19:26:29.911029771 +0000 UTC m=+30.786576451" Jul 06 19:26:30.505353 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:30.505316 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sv6xg"] Jul 06 19:26:30.505550 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:30.505459 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:30.505620 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:30.505577 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:30.507101 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:30.507074 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m64lw"] Jul 06 19:26:30.507204 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:30.507167 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:30.507263 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:30.507238 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:30.509915 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:30.509893 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cs4pr"] Jul 06 19:26:30.510035 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:30.509970 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:30.510099 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:30.510056 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:30.881120 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:30.881042 2545 generic.go:356] "Generic (PLEG): container finished" podID="14d0d201-5971-445b-80f5-c4addff50780" containerID="f6044904b917189580ac446ea743d438d122ec79dd850d667e302d6c1ddc4937" exitCode=0 Jul 06 19:26:30.881575 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:30.881130 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerDied","Data":"f6044904b917189580ac446ea743d438d122ec79dd850d667e302d6c1ddc4937"} Jul 06 19:26:31.886182 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:31.886008 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerStarted","Data":"075e62858481f96b95d7660ad9f49323f2237c3dc840cb54ad5e86689273ad24"} Jul 06 19:26:32.598942 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:32.598881 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:32.599065 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:32.598986 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:32.599065 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:32.598990 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:32.599065 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:32.599013 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:32.599160 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:32.599091 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:32.599191 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:32.599157 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:32.891104 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:32.891077 2545 generic.go:356] "Generic (PLEG): container finished" podID="14d0d201-5971-445b-80f5-c4addff50780" containerID="075e62858481f96b95d7660ad9f49323f2237c3dc840cb54ad5e86689273ad24" exitCode=0 Jul 06 19:26:32.891481 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:32.891115 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerDied","Data":"075e62858481f96b95d7660ad9f49323f2237c3dc840cb54ad5e86689273ad24"} Jul 06 19:26:33.278100 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:33.278014 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:33.278100 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:33.278067 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:33.278278 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:33.278136 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:33.278278 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:33.278136 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:33.278278 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:33.278181 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:05.278167256 +0000 UTC m=+66.153713915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:26:33.278278 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:33.278193 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret podName:fdd867de-45fa-4f74-9908-1ff7cb1314e9 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:05.278187377 +0000 UTC m=+66.153734035 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret") pod "global-pull-secret-syncer-cs4pr" (UID: "fdd867de-45fa-4f74-9908-1ff7cb1314e9") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:26:33.480429 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:33.480375 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:33.480606 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:33.480531 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:26:33.480606 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:33.480548 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:26:33.480606 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:33.480557 2545 projected.go:196] Error preparing data for projected volume kube-api-access-gklk4 for pod openshift-network-diagnostics/network-check-target-m64lw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:33.480606 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:33.480605 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4 podName:49f0b1f7-bd6e-4377-b5ca-896c1d077438 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:05.480589676 +0000 UTC m=+66.356136354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gklk4" (UniqueName: "kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4") pod "network-check-target-m64lw" (UID: "49f0b1f7-bd6e-4377-b5ca-896c1d077438") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:26:34.598112 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:34.598060 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:34.598633 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:34.598179 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:34.598633 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:34.598190 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cs4pr" podUID="fdd867de-45fa-4f74-9908-1ff7cb1314e9" Jul 06 19:26:34.598633 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:34.598406 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:34.598633 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:34.598540 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m64lw" podUID="49f0b1f7-bd6e-4377-b5ca-896c1d077438" Jul 06 19:26:34.598788 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:34.598647 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:26:34.974889 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:34.974861 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeReady" Jul 06 19:26:34.975049 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:34.974981 2545 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:26:35.018847 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.018809 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg"] Jul 06 19:26:35.020786 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.020764 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.022559 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.022531 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct"] Jul 06 19:26:35.023481 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.023459 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:26:35.023611 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.023507 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:26:35.023838 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.023818 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:26:35.024251 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.024230 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 19:26:35.024886 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.024863 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.026525 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.026503 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-4gj67"] Jul 06 19:26:35.027907 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.027891 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw"] Jul 06 19:26:35.028010 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.027997 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:35.028965 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.028945 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 19:26:35.029068 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.029036 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 19:26:35.029328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.029270 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 19:26:35.029450 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.029384 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8545644b6-jngbb"] Jul 06 19:26:35.029450 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.029436 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 19:26:35.030356 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.029651 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:26:35.030356 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.030147 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 19:26:35.030356 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.030352 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 19:26:35.031171 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.031145 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.031959 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.031940 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 19:26:35.032049 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.031950 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-s9hqf\"" Jul 06 19:26:35.034124 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.034107 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5mh4g\"" Jul 06 19:26:35.034605 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.034589 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:26:35.035567 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.035534 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cvbvc\"" Jul 06 19:26:35.035724 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.035695 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:26:35.035896 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.035877 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:26:35.036627 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.036610 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sqdtt"] Jul 06 19:26:35.039898 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.039441 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg"] Jul 06 19:26:35.039898 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.039561 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.043451 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.040333 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw"] Jul 06 19:26:35.043699 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.043652 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:26:35.044026 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.043827 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:26:35.044026 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.043927 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7f4qs\"" Jul 06 19:26:35.046329 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.046311 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d6m7w"] Jul 06 19:26:35.047615 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.047599 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:26:35.048143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.048126 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d6m7w" Jul 06 19:26:35.050462 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.050446 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:26:35.050541 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.050497 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:26:35.050823 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.050799 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kkk98\"" Jul 06 19:26:35.051104 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.051082 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:26:35.052038 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.052022 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8545644b6-jngbb"] Jul 06 19:26:35.056069 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.056048 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-4gj67"] Jul 06 19:26:35.059460 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.059392 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct"] Jul 06 19:26:35.064237 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.064206 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d6m7w"] Jul 06 19:26:35.072410 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.072391 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqdtt"] Jul 06 19:26:35.195149 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195125 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-ca\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.195288 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195165 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4s58\" (UniqueName: \"kubernetes.io/projected/c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6-kube-api-access-s4s58\") pod \"managed-serviceaccount-addon-agent-859566fb64-mjhsw\" (UID: \"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:26:35.195288 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195203 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-image-registry-private-configuration\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.195288 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195228 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-certificates\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.195288 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195250 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:35.195288 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195266 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-hub\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195306 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195326 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-installation-pull-secrets\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195347 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xc7b\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-kube-api-access-5xc7b\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195375 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c720a33-92fe-4330-8d54-f2ed209f785d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195396 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195453 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvbp\" (UniqueName: \"kubernetes.io/projected/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-kube-api-access-9cvbp\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195468 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-trusted-ca\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195510 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b877b8c-c234-4d29-aa5f-078c5de3726e-tmp-dir\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.195547 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195541 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195559 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-bound-sa-token\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195586 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfmx\" (UniqueName: \"kubernetes.io/projected/4c720a33-92fe-4330-8d54-f2ed209f785d-kube-api-access-tzfmx\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195619 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-tmp\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195650 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-ca-trust-extracted\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195673 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195715 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-859566fb64-mjhsw\" (UID: \"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195737 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c05a8cd-b758-43e3-9259-0ef57bcd2b6a-cert\") pod \"ingress-canary-d6m7w\" (UID: \"5c05a8cd-b758-43e3-9259-0ef57bcd2b6a\") " pod="openshift-ingress-canary/ingress-canary-d6m7w" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195765 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b877b8c-c234-4d29-aa5f-078c5de3726e-config-volume\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195787 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph68d\" (UniqueName: \"kubernetes.io/projected/5c05a8cd-b758-43e3-9259-0ef57bcd2b6a-kube-api-access-ph68d\") pod \"ingress-canary-d6m7w\" (UID: \"5c05a8cd-b758-43e3-9259-0ef57bcd2b6a\") " pod="openshift-ingress-canary/ingress-canary-d6m7w" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195831 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195847 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrhf\" (UniqueName: \"kubernetes.io/projected/9b877b8c-c234-4d29-aa5f-078c5de3726e-kube-api-access-vmrhf\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.195911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195864 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-klusterlet-config\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.196239 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.195935 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:35.296777 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.296701 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-859566fb64-mjhsw\" (UID: \"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:26:35.296777 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.296745 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c05a8cd-b758-43e3-9259-0ef57bcd2b6a-cert\") pod \"ingress-canary-d6m7w\" (UID: \"5c05a8cd-b758-43e3-9259-0ef57bcd2b6a\") " pod="openshift-ingress-canary/ingress-canary-d6m7w" Jul 06 19:26:35.296777 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.296773 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b877b8c-c234-4d29-aa5f-078c5de3726e-config-volume\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.297028 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.296797 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph68d\" (UniqueName: \"kubernetes.io/projected/5c05a8cd-b758-43e3-9259-0ef57bcd2b6a-kube-api-access-ph68d\") pod \"ingress-canary-d6m7w\" (UID: \"5c05a8cd-b758-43e3-9259-0ef57bcd2b6a\") " pod="openshift-ingress-canary/ingress-canary-d6m7w" Jul 06 19:26:35.297028 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.296824 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.297028 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.296927 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:26:35.297028 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.296941 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8545644b6-jngbb: secret "image-registry-tls" not found Jul 06 19:26:35.297028 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.296996 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls podName:c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:35.796977554 +0000 UTC m=+36.672524226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls") pod "image-registry-8545644b6-jngbb" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2") : secret "image-registry-tls" not found Jul 06 19:26:35.297376 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297340 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b877b8c-c234-4d29-aa5f-078c5de3726e-config-volume\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.297519 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297401 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrhf\" (UniqueName: \"kubernetes.io/projected/9b877b8c-c234-4d29-aa5f-078c5de3726e-kube-api-access-vmrhf\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.297519 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297463 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-klusterlet-config\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.297519 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297497 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-ca\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.297658 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297560 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4s58\" (UniqueName: \"kubernetes.io/projected/c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6-kube-api-access-s4s58\") pod \"managed-serviceaccount-addon-agent-859566fb64-mjhsw\" (UID: \"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:26:35.297658 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297587 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-image-registry-private-configuration\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.297658 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297609 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-certificates\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.297658 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297638 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:35.297658 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297657 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-hub\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.297843 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297693 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.297843 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297715 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-installation-pull-secrets\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.297843 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297734 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xc7b\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-kube-api-access-5xc7b\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.297843 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297759 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c720a33-92fe-4330-8d54-f2ed209f785d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.297843 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297777 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.297843 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297819 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvbp\" (UniqueName: \"kubernetes.io/projected/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-kube-api-access-9cvbp\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297838 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-trusted-ca\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297874 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b877b8c-c234-4d29-aa5f-078c5de3726e-tmp-dir\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297901 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297920 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-bound-sa-token\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297945 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfmx\" (UniqueName: \"kubernetes.io/projected/4c720a33-92fe-4330-8d54-f2ed209f785d-kube-api-access-tzfmx\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297968 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-tmp\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.297988 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-ca-trust-extracted\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.298013 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:35.298110 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.298074 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:26:35.298526 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.298127 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls podName:9b877b8c-c234-4d29-aa5f-078c5de3726e nodeName:}" failed. No retries permitted until 2026-07-06 19:26:35.798111647 +0000 UTC m=+36.673658322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls") pod "dns-default-sqdtt" (UID: "9b877b8c-c234-4d29-aa5f-078c5de3726e") : secret "dns-default-metrics-tls" not found Jul 06 19:26:35.298693 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.298671 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:35.299720 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.299289 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:26:35.299720 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.299411 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert podName:2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b nodeName:}" failed. No retries permitted until 2026-07-06 19:26:35.799395467 +0000 UTC m=+36.674942126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4gj67" (UID: "2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b") : secret "networking-console-plugin-cert" not found Jul 06 19:26:35.300026 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.300004 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-tmp\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.300359 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.300335 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-ca-trust-extracted\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.300654 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.300633 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-trusted-ca\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.301024 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.301000 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-certificates\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.302679 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.302625 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b877b8c-c234-4d29-aa5f-078c5de3726e-tmp-dir\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.303126 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.302968 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c05a8cd-b758-43e3-9259-0ef57bcd2b6a-cert\") pod \"ingress-canary-d6m7w\" (UID: \"5c05a8cd-b758-43e3-9259-0ef57bcd2b6a\") " pod="openshift-ingress-canary/ingress-canary-d6m7w" Jul 06 19:26:35.303126 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.302972 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-klusterlet-config\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.303126 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.303119 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c720a33-92fe-4330-8d54-f2ed209f785d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.303311 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.303163 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.303311 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.303234 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-ca\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.303434 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.303362 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-859566fb64-mjhsw\" (UID: \"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:26:35.305231 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.304797 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-image-registry-private-configuration\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.305231 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.304901 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-installation-pull-secrets\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.305231 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.305103 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:35.305468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.305341 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.306010 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.305988 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4c720a33-92fe-4330-8d54-f2ed209f785d-hub\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.308314 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.308229 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrhf\" (UniqueName: \"kubernetes.io/projected/9b877b8c-c234-4d29-aa5f-078c5de3726e-kube-api-access-vmrhf\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.308406 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.308359 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-bound-sa-token\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.308999 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.308975 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4s58\" (UniqueName: \"kubernetes.io/projected/c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6-kube-api-access-s4s58\") pod \"managed-serviceaccount-addon-agent-859566fb64-mjhsw\" (UID: \"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:26:35.309533 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.309512 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfmx\" (UniqueName: \"kubernetes.io/projected/4c720a33-92fe-4330-8d54-f2ed209f785d-kube-api-access-tzfmx\") pod \"cluster-proxy-proxy-agent-7cd8c5f557-fvbct\" (UID: \"4c720a33-92fe-4330-8d54-f2ed209f785d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.310046 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.309901 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvbp\" (UniqueName: \"kubernetes.io/projected/95ca4ab3-d09e-4019-9a9c-7d31dfd509ab-kube-api-access-9cvbp\") pod \"klusterlet-addon-workmgr-546866d9f-jgcqg\" (UID: \"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.310941 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.310919 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xc7b\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-kube-api-access-5xc7b\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.311084 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.311066 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph68d\" (UniqueName: \"kubernetes.io/projected/5c05a8cd-b758-43e3-9259-0ef57bcd2b6a-kube-api-access-ph68d\") pod \"ingress-canary-d6m7w\" (UID: \"5c05a8cd-b758-43e3-9259-0ef57bcd2b6a\") " pod="openshift-ingress-canary/ingress-canary-d6m7w" Jul 06 19:26:35.334001 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.333973 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:35.356064 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.356033 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:26:35.372737 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.372709 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:26:35.410999 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.410542 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d6m7w" Jul 06 19:26:35.545967 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.545892 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg"] Jul 06 19:26:35.553331 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:35.553298 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ca4ab3_d09e_4019_9a9c_7d31dfd509ab.slice/crio-b7f1087af5dbd25fac3f605fbe571ea6bc9ccfcf2b1de2cd71e48a98a7ab596d WatchSource:0}: Error finding container b7f1087af5dbd25fac3f605fbe571ea6bc9ccfcf2b1de2cd71e48a98a7ab596d: Status 404 returned error can't find the container with id b7f1087af5dbd25fac3f605fbe571ea6bc9ccfcf2b1de2cd71e48a98a7ab596d Jul 06 19:26:35.562120 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.562095 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct"] Jul 06 19:26:35.565323 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:35.565301 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c720a33_92fe_4330_8d54_f2ed209f785d.slice/crio-99265e1981ea2bdf94d33528a8b5b840e5decd2c97584aa2180d2f617999b1ad WatchSource:0}: Error finding container 99265e1981ea2bdf94d33528a8b5b840e5decd2c97584aa2180d2f617999b1ad: Status 404 returned error can't find the container with id 99265e1981ea2bdf94d33528a8b5b840e5decd2c97584aa2180d2f617999b1ad Jul 06 19:26:35.587399 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.587376 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw"] Jul 06 19:26:35.590956 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:35.590935 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc97abb00_0cfe_4e3a_a181_e0da7bcb3ce6.slice/crio-f5df0d94be3a98694d7b5b06dd787c156db65ba354c155c9b9a9aeccc43d3843 WatchSource:0}: Error finding container f5df0d94be3a98694d7b5b06dd787c156db65ba354c155c9b9a9aeccc43d3843: Status 404 returned error can't find the container with id f5df0d94be3a98694d7b5b06dd787c156db65ba354c155c9b9a9aeccc43d3843 Jul 06 19:26:35.595227 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.594724 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d6m7w"] Jul 06 19:26:35.598656 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:26:35.598628 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c05a8cd_b758_43e3_9259_0ef57bcd2b6a.slice/crio-bd1d446ee39c02c4b965a789ed84e3772672f18e3332cc6dc4c8fc691165955b WatchSource:0}: Error finding container bd1d446ee39c02c4b965a789ed84e3772672f18e3332cc6dc4c8fc691165955b: Status 404 returned error can't find the container with id bd1d446ee39c02c4b965a789ed84e3772672f18e3332cc6dc4c8fc691165955b Jul 06 19:26:35.801303 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.801215 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:35.801303 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.801263 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:35.801524 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.801322 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:35.801524 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.801371 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:26:35.801524 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.801444 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:26:35.801524 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.801447 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:26:35.801524 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.801464 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8545644b6-jngbb: secret "image-registry-tls" not found Jul 06 19:26:35.801524 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.801465 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert podName:2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b nodeName:}" failed. No retries permitted until 2026-07-06 19:26:36.801448153 +0000 UTC m=+37.676994815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4gj67" (UID: "2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b") : secret "networking-console-plugin-cert" not found Jul 06 19:26:35.801524 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.801507 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls podName:9b877b8c-c234-4d29-aa5f-078c5de3726e nodeName:}" failed. No retries permitted until 2026-07-06 19:26:36.801490979 +0000 UTC m=+37.677037645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls") pod "dns-default-sqdtt" (UID: "9b877b8c-c234-4d29-aa5f-078c5de3726e") : secret "dns-default-metrics-tls" not found Jul 06 19:26:35.801524 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:35.801523 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls podName:c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:36.801513462 +0000 UTC m=+37.677060120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls") pod "image-registry-8545644b6-jngbb" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2") : secret "image-registry-tls" not found Jul 06 19:26:35.900231 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.900188 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" event={"ID":"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab","Type":"ContainerStarted","Data":"b7f1087af5dbd25fac3f605fbe571ea6bc9ccfcf2b1de2cd71e48a98a7ab596d"} Jul 06 19:26:35.901397 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.901365 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" event={"ID":"4c720a33-92fe-4330-8d54-f2ed209f785d","Type":"ContainerStarted","Data":"99265e1981ea2bdf94d33528a8b5b840e5decd2c97584aa2180d2f617999b1ad"} Jul 06 19:26:35.902692 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.902639 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d6m7w" event={"ID":"5c05a8cd-b758-43e3-9259-0ef57bcd2b6a","Type":"ContainerStarted","Data":"bd1d446ee39c02c4b965a789ed84e3772672f18e3332cc6dc4c8fc691165955b"} Jul 06 19:26:35.903721 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:35.903685 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" event={"ID":"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6","Type":"ContainerStarted","Data":"f5df0d94be3a98694d7b5b06dd787c156db65ba354c155c9b9a9aeccc43d3843"} Jul 06 19:26:36.598245 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.598212 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:26:36.600031 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.598668 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:26:36.600031 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.599043 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:26:36.602553 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.602534 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:26:36.602749 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.602736 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:26:36.602923 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.602903 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:26:36.603040 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.603028 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:26:36.603338 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.603315 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5qdm\"" Jul 06 19:26:36.606131 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.603552 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdwtt\"" Jul 06 19:26:36.810995 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.810956 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:36.811148 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.811033 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:36.811148 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:36.811112 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:36.811312 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:36.811292 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:26:36.811376 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:36.811365 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls podName:9b877b8c-c234-4d29-aa5f-078c5de3726e nodeName:}" failed. No retries permitted until 2026-07-06 19:26:38.811345641 +0000 UTC m=+39.686892304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls") pod "dns-default-sqdtt" (UID: "9b877b8c-c234-4d29-aa5f-078c5de3726e") : secret "dns-default-metrics-tls" not found Jul 06 19:26:36.811834 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:36.811794 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:26:36.811834 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:36.811812 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8545644b6-jngbb: secret "image-registry-tls" not found Jul 06 19:26:36.811948 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:36.811856 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls podName:c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:38.811840087 +0000 UTC m=+39.687386752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls") pod "image-registry-8545644b6-jngbb" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2") : secret "image-registry-tls" not found Jul 06 19:26:36.811948 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:36.811918 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:26:36.812067 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:36.812027 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert podName:2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b nodeName:}" failed. No retries permitted until 2026-07-06 19:26:38.811939785 +0000 UTC m=+39.687486457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4gj67" (UID: "2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b") : secret "networking-console-plugin-cert" not found Jul 06 19:26:37.444227 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:37.444176 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:38.831874 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:38.831842 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:38.831954 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:38.832001 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:38.832024 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:38.832097 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:38.832101 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:38.832117 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8545644b6-jngbb: secret "image-registry-tls" not found Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:38.832106 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls podName:9b877b8c-c234-4d29-aa5f-078c5de3726e nodeName:}" failed. No retries permitted until 2026-07-06 19:26:42.832082807 +0000 UTC m=+43.707629481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls") pod "dns-default-sqdtt" (UID: "9b877b8c-c234-4d29-aa5f-078c5de3726e") : secret "dns-default-metrics-tls" not found Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:38.832187 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert podName:2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b nodeName:}" failed. No retries permitted until 2026-07-06 19:26:42.832156786 +0000 UTC m=+43.707703467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4gj67" (UID: "2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b") : secret "networking-console-plugin-cert" not found Jul 06 19:26:38.832333 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:38.832207 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls podName:c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:42.832195249 +0000 UTC m=+43.707741912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls") pod "image-registry-8545644b6-jngbb" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2") : secret "image-registry-tls" not found Jul 06 19:26:40.307801 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:40.307728 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:40.375662 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:40.375630 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:40.534260 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:40.534219 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:40.634455 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:40.634354 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:42.867140 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:42.867099 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:42.867187 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:42.867216 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:42.867232 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8545644b6-jngbb: secret "image-registry-tls" not found Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:42.867238 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:42.867303 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls podName:c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:50.867284575 +0000 UTC m=+51.742831234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls") pod "image-registry-8545644b6-jngbb" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2") : secret "image-registry-tls" not found Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:42.867345 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:42.867348 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:42.867400 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls podName:9b877b8c-c234-4d29-aa5f-078c5de3726e nodeName:}" failed. No retries permitted until 2026-07-06 19:26:50.867389533 +0000 UTC m=+51.742936206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls") pod "dns-default-sqdtt" (UID: "9b877b8c-c234-4d29-aa5f-078c5de3726e") : secret "dns-default-metrics-tls" not found Jul 06 19:26:42.867613 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:42.867434 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert podName:2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b nodeName:}" failed. No retries permitted until 2026-07-06 19:26:50.867407951 +0000 UTC m=+51.742954635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4gj67" (UID: "2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b") : secret "networking-console-plugin-cert" not found Jul 06 19:26:43.705001 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:43.704973 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:43.803930 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:43.803901 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:44.932711 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.932677 2545 generic.go:356] "Generic (PLEG): container finished" podID="14d0d201-5971-445b-80f5-c4addff50780" containerID="4010ddbcadcbcbc4a569d4f661a758ffa19b0ddde9ee460917fc16d8d7ca7eba" exitCode=0 Jul 06 19:26:44.933091 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.932720 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerDied","Data":"4010ddbcadcbcbc4a569d4f661a758ffa19b0ddde9ee460917fc16d8d7ca7eba"} Jul 06 19:26:44.934262 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.934240 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" event={"ID":"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab","Type":"ContainerStarted","Data":"c11aabedee5f0e1a64fa499cf46f8d649f3bf5f2e151490b8138eb057bcd045c"} Jul 06 19:26:44.934407 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.934394 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:44.936223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.936207 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:26:44.936526 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.936504 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" event={"ID":"4c720a33-92fe-4330-8d54-f2ed209f785d","Type":"ContainerStarted","Data":"23da1ee4dd5a0000cfe42e370e1222f68819634a86785d7805d7f95f36216d52"} Jul 06 19:26:44.936635 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.936538 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" event={"ID":"4c720a33-92fe-4330-8d54-f2ed209f785d","Type":"ContainerStarted","Data":"df0ad700fbeb57e6049b0be82b01a4b6f0b2c762985a12a975a431a32ecfde31"} Jul 06 19:26:44.936635 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.936555 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" event={"ID":"4c720a33-92fe-4330-8d54-f2ed209f785d","Type":"ContainerStarted","Data":"3fad5685b2b579912c090561aee4fa7c2d2806b4bd56df1f4da501013af4458a"} Jul 06 19:26:44.937933 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.937910 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d6m7w" event={"ID":"5c05a8cd-b758-43e3-9259-0ef57bcd2b6a","Type":"ContainerStarted","Data":"d962017e30ebeeef8f0fb3302509995c246b03dc20e650214839f68a9563b1b9"} Jul 06 19:26:44.939268 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.939246 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" event={"ID":"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6","Type":"ContainerStarted","Data":"f30d263a4bfd43e5629c2271bcf04fec23fbbfa44fb9dc89a1c09bcf34eb39f2"} Jul 06 19:26:44.980872 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.980833 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" podStartSLOduration=8.542236971 podStartE2EDuration="16.980820126s" totalImagesPullingTime="8.438583155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:44.980499579 +0000 UTC m=+45.856046264" watchObservedRunningTime="2026-07-06 19:26:44.980820126 +0000 UTC m=+45.856366830" Jul 06 19:26:45.000115 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:44.999701 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" podStartSLOduration=8.542358052 podStartE2EDuration="16.999656035s" totalImagesPullingTime="8.457297983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:44.998380311 +0000 UTC m=+45.873926991" watchObservedRunningTime="2026-07-06 19:26:44.999656035 +0000 UTC m=+45.875202717" Jul 06 19:26:45.020528 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:45.020490 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" podStartSLOduration=8.523379350999999 podStartE2EDuration="17.020476052s" totalImagesPullingTime="8.497096701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:45.020006144 +0000 UTC m=+45.895552825" watchObservedRunningTime="2026-07-06 19:26:45.020476052 +0000 UTC m=+45.896022732" Jul 06 19:26:45.037889 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:45.037784 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d6m7w" podStartSLOduration=1.616886518 podStartE2EDuration="10.037771028s" totalImagesPullingTime="8.42088451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:45.036708154 +0000 UTC m=+45.912254845" watchObservedRunningTime="2026-07-06 19:26:45.037771028 +0000 UTC m=+45.913317721" Jul 06 19:26:45.945157 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:45.945124 2545 generic.go:356] "Generic (PLEG): container finished" podID="14d0d201-5971-445b-80f5-c4addff50780" containerID="2b35c41be87811af0ade64f5c1f783e7fbec78ea6dce102b11fe8ea1599c1217" exitCode=0 Jul 06 19:26:45.945560 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:45.945225 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerDied","Data":"2b35c41be87811af0ade64f5c1f783e7fbec78ea6dce102b11fe8ea1599c1217"} Jul 06 19:26:46.952512 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:46.952470 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" event={"ID":"14d0d201-5971-445b-80f5-c4addff50780","Type":"ContainerStarted","Data":"36f92c643ceff6c3eb02bf872908ba841637808be5baaf134a912893e6e51b38"} Jul 06 19:26:46.988357 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:46.988311 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m7v7k" podStartSLOduration=8.662901023 podStartE2EDuration="46.988297871s" totalImagesPullingTime="38.325396848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:46.987827508 +0000 UTC m=+47.863374189" watchObservedRunningTime="2026-07-06 19:26:46.988297871 +0000 UTC m=+47.863844552" Jul 06 19:26:47.230190 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:47.230109 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:47.382377 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:47.382347 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:48.007579 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:48.007552 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:48.110001 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:48.109963 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:49.549163 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:49.549132 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:49.623331 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:49.623304 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:50.834981 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:50.834949 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:50.931748 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:50.931718 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:26:50.931878 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:50.931766 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:26:50.931878 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:50.931794 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:26:50.931878 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:50.931863 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:26:50.932057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:50.931885 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:26:50.932057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:50.931887 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8545644b6-jngbb: secret "image-registry-tls" not found Jul 06 19:26:50.932057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:50.931930 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:26:50.932057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:50.931939 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls podName:9b877b8c-c234-4d29-aa5f-078c5de3726e nodeName:}" failed. No retries permitted until 2026-07-06 19:27:06.931926802 +0000 UTC m=+67.807473461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls") pod "dns-default-sqdtt" (UID: "9b877b8c-c234-4d29-aa5f-078c5de3726e") : secret "dns-default-metrics-tls" not found Jul 06 19:26:50.932057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:50.931964 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls podName:c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:06.93194739 +0000 UTC m=+67.807494067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls") pod "image-registry-8545644b6-jngbb" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2") : secret "image-registry-tls" not found Jul 06 19:26:50.932057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:26:50.931984 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert podName:2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b nodeName:}" failed. No retries permitted until 2026-07-06 19:27:06.931973777 +0000 UTC m=+67.807520437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4gj67" (UID: "2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b") : secret "networking-console-plugin-cert" not found Jul 06 19:26:50.932590 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:50.932571 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:54.010117 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:54.010087 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:54.172729 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:54.172699 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:54.277234 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:54.277167 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:54.366975 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:54.366951 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:55.082790 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:55.082761 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:26:55.165763 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:26:55.165724 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:01.269358 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:01.269325 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:01.896495 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:01.896469 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mcmq" Jul 06 19:27:02.087397 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:02.087365 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:02.163112 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:02.163045 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:02.246676 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:02.246644 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:05.347329 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.347295 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:27:05.347702 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.347339 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:27:05.349730 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.349714 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:27:05.349864 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.349850 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:27:05.358446 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:05.358409 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:27:05.358573 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:05.358480 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:09.35846409 +0000 UTC m=+130.234010749 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : secret "metrics-daemon-secret" not found Jul 06 19:27:05.361732 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.361706 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdd867de-45fa-4f74-9908-1ff7cb1314e9-original-pull-secret\") pod \"global-pull-secret-syncer-cs4pr\" (UID: \"fdd867de-45fa-4f74-9908-1ff7cb1314e9\") " pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:27:05.429962 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.429934 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cs4pr" Jul 06 19:27:05.548893 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.548865 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:27:05.551356 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.551334 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:27:05.551803 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.551784 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cs4pr"] Jul 06 19:27:05.554636 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:27:05.554609 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd867de_45fa_4f74_9908_1ff7cb1314e9.slice/crio-ff7f097c53b7e770b1dd1be8302d146cae8c4c2436bc955244845dadc59a5115 WatchSource:0}: Error finding container ff7f097c53b7e770b1dd1be8302d146cae8c4c2436bc955244845dadc59a5115: Status 404 returned error can't find the container with id ff7f097c53b7e770b1dd1be8302d146cae8c4c2436bc955244845dadc59a5115 Jul 06 19:27:05.562012 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.561987 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:27:05.576140 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.576116 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gklk4\" (UniqueName: \"kubernetes.io/projected/49f0b1f7-bd6e-4377-b5ca-896c1d077438-kube-api-access-gklk4\") pod \"network-check-target-m64lw\" (UID: \"49f0b1f7-bd6e-4377-b5ca-896c1d077438\") " pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:27:05.720040 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.720015 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdwtt\"" Jul 06 19:27:05.728074 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.728054 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:27:05.848309 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:05.848275 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m64lw"] Jul 06 19:27:05.850776 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:27:05.850751 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f0b1f7_bd6e_4377_b5ca_896c1d077438.slice/crio-7e91a35fa94aa3b0ebcf16026e5e66c8df68876aae2d7636e752fa01b6935914 WatchSource:0}: Error finding container 7e91a35fa94aa3b0ebcf16026e5e66c8df68876aae2d7636e752fa01b6935914: Status 404 returned error can't find the container with id 7e91a35fa94aa3b0ebcf16026e5e66c8df68876aae2d7636e752fa01b6935914 Jul 06 19:27:06.008027 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:06.007952 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m64lw" event={"ID":"49f0b1f7-bd6e-4377-b5ca-896c1d077438","Type":"ContainerStarted","Data":"7e91a35fa94aa3b0ebcf16026e5e66c8df68876aae2d7636e752fa01b6935914"} Jul 06 19:27:06.008903 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:06.008882 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cs4pr" event={"ID":"fdd867de-45fa-4f74-9908-1ff7cb1314e9","Type":"ContainerStarted","Data":"ff7f097c53b7e770b1dd1be8302d146cae8c4c2436bc955244845dadc59a5115"} Jul 06 19:27:06.153878 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:06.153837 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:06.960466 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:06.960429 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:06.960499 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:06.960561 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:06.960609 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:06.960619 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:06.960631 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8545644b6-jngbb: secret "image-registry-tls" not found Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:06.960699 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls podName:c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:38.960680342 +0000 UTC m=+99.836227006 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls") pod "image-registry-8545644b6-jngbb" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2") : secret "image-registry-tls" not found Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:06.960727 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert podName:2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b nodeName:}" failed. No retries permitted until 2026-07-06 19:27:38.960708943 +0000 UTC m=+99.836255628 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4gj67" (UID: "2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b") : secret "networking-console-plugin-cert" not found Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:06.960785 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:06.961057 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:06.960830 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls podName:9b877b8c-c234-4d29-aa5f-078c5de3726e nodeName:}" failed. No retries permitted until 2026-07-06 19:27:38.960815537 +0000 UTC m=+99.836362199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls") pod "dns-default-sqdtt" (UID: "9b877b8c-c234-4d29-aa5f-078c5de3726e") : secret "dns-default-metrics-tls" not found Jul 06 19:27:10.520060 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:10.520019 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:10.590588 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:10.590562 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:11.025404 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:11.025365 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m64lw" event={"ID":"49f0b1f7-bd6e-4377-b5ca-896c1d077438","Type":"ContainerStarted","Data":"0dedb057627759a4e04cd823f6b91f0edeecb64097450552c75f62dc184418d9"} Jul 06 19:27:11.025594 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:11.025435 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:27:11.026735 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:11.026711 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cs4pr" event={"ID":"fdd867de-45fa-4f74-9908-1ff7cb1314e9","Type":"ContainerStarted","Data":"5c05727a0f40fa6f0c3d16ae68faa91b6bc65eb5b3e6daacd455b57347f174ba"} Jul 06 19:27:11.044619 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:11.044572 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m64lw" podStartSLOduration=66.872858661 podStartE2EDuration="1m11.044557682s" totalImagesPullingTime="4.171699021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:11.043548479 +0000 UTC m=+71.919095159" watchObservedRunningTime="2026-07-06 19:27:11.044557682 +0000 UTC m=+71.920104354" Jul 06 19:27:11.062321 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:11.062282 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cs4pr" podStartSLOduration=66.589239918 podStartE2EDuration="1m11.062268314s" totalImagesPullingTime="4.473028396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:11.061209339 +0000 UTC m=+71.936756020" watchObservedRunningTime="2026-07-06 19:27:11.062268314 +0000 UTC m=+71.937814995" Jul 06 19:27:11.617151 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:11.617122 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:11.718373 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:11.718348 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:13.178831 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:13.178794 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:13.278280 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:13.278252 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:32.123819 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:32.123706 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:32.192836 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:32.192808 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:27:38.993236 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:38.993180 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:27:38.993236 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:38.993244 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:27:38.993832 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:38.993273 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:27:38.993832 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:38.993345 2545 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:27:38.993832 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:38.993364 2545 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8545644b6-jngbb: secret "image-registry-tls" not found Jul 06 19:27:38.993832 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:38.993382 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:27:38.993832 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:38.993390 2545 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:27:38.993832 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:38.993452 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls podName:c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2 nodeName:}" failed. No retries permitted until 2026-07-06 19:28:42.993437125 +0000 UTC m=+163.868983796 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls") pod "image-registry-8545644b6-jngbb" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2") : secret "image-registry-tls" not found Jul 06 19:27:38.993832 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:38.993465 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls podName:9b877b8c-c234-4d29-aa5f-078c5de3726e nodeName:}" failed. No retries permitted until 2026-07-06 19:28:42.993459285 +0000 UTC m=+163.869005943 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls") pod "dns-default-sqdtt" (UID: "9b877b8c-c234-4d29-aa5f-078c5de3726e") : secret "dns-default-metrics-tls" not found Jul 06 19:27:38.993832 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:27:38.993489 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert podName:2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b nodeName:}" failed. No retries permitted until 2026-07-06 19:28:42.993470099 +0000 UTC m=+163.869016778 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4gj67" (UID: "2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b") : secret "networking-console-plugin-cert" not found Jul 06 19:27:42.032261 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:27:42.032235 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m64lw" Jul 06 19:28:09.420709 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:09.420670 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:28:09.421123 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:09.420788 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:28:09.421123 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:09.420841 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs podName:ed68dfa2-ff3c-4f7c-bf88-414559a5c413 nodeName:}" failed. No retries permitted until 2026-07-06 19:30:11.420827015 +0000 UTC m=+252.296373674 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs") pod "network-metrics-daemon-sv6xg" (UID: "ed68dfa2-ff3c-4f7c-bf88-414559a5c413") : secret "metrics-daemon-secret" not found Jul 06 19:28:13.006588 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:13.006556 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:14.169584 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:14.169551 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:16.475884 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.475855 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nzb5n"] Jul 06 19:28:16.477788 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.477773 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.488288 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:16.488261 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" Jul 06 19:28:16.498564 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:16.498541 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 06 19:28:16.498672 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:16.498610 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 06 19:28:16.498672 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:16.498634 2545 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-nzb5n\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="5e62983e-ae49-42d3-b0c1-293fee95364f" pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.498766 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:16.498702 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-k82dj\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k82dj\"" type="*v1.Secret" Jul 06 19:28:16.498807 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:16.498755 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" Jul 06 19:28:16.516250 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.516227 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nzb5n"] Jul 06 19:28:16.565649 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.565617 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:16.573660 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.573634 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.573758 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.573670 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ms2s\" (UniqueName: \"kubernetes.io/projected/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-api-access-8ms2s\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.573758 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.573705 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e62983e-ae49-42d3-b0c1-293fee95364f-crio-socket\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.573758 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.573755 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e62983e-ae49-42d3-b0c1-293fee95364f-data-volume\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.573850 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.573771 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.674202 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.674167 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e62983e-ae49-42d3-b0c1-293fee95364f-data-volume\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.674202 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.674203 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.674345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.674245 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.674345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.674264 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ms2s\" (UniqueName: \"kubernetes.io/projected/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-api-access-8ms2s\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.674345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.674287 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e62983e-ae49-42d3-b0c1-293fee95364f-crio-socket\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.674469 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.674376 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e62983e-ae49-42d3-b0c1-293fee95364f-crio-socket\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.674565 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.674547 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e62983e-ae49-42d3-b0c1-293fee95364f-data-volume\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:16.746443 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:16.746315 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:17.505041 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:17.505009 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:17.548618 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:17.548586 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:28:17.675308 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:17.675271 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: failed to sync configmap cache: timed out waiting for the condition Jul 06 19:28:17.675495 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:17.675271 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition Jul 06 19:28:17.675495 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:17.675367 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:18.175349775 +0000 UTC m=+139.050896434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : failed to sync configmap cache: timed out waiting for the condition Jul 06 19:28:17.675495 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:17.675466 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:18.175444368 +0000 UTC m=+139.050991031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : failed to sync secret cache: timed out waiting for the condition Jul 06 19:28:17.740071 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:17.740042 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:17.767819 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:17.767755 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:28:17.929245 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:17.929215 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:28:17.953847 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:17.953817 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:28:17.964199 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:17.964179 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ms2s\" (UniqueName: \"kubernetes.io/projected/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-api-access-8ms2s\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:18.101864 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:18.101778 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k82dj\"" Jul 06 19:28:18.184545 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:18.184515 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:18.184661 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:18.184563 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:18.184661 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:18.184636 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:18.184727 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:18.184659 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:18.184727 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:18.184707 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:19.18469052 +0000 UTC m=+140.060237180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:18.184727 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:18.184723 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:19.184714881 +0000 UTC m=+140.060261539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:18.642041 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:18.642011 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tcfl6_0dd9923b-0994-4f34-a64b-fb5de561aefd/dns-node-resolver/0.log" Jul 06 19:28:19.193597 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:19.193549 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:19.193801 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:19.193668 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:19.193801 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:19.193724 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:19.193801 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:19.193750 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:19.193988 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:19.193812 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:21.193788386 +0000 UTC m=+142.069335060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:19.193988 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:19.193834 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:21.19382365 +0000 UTC m=+142.069370309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:19.244665 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:19.244642 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lz9j9_a211f40a-7213-4ea6-9716-7b8e1a139c9e/node-ca/0.log" Jul 06 19:28:20.049075 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:20.049048 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d6m7w_5c05a8cd-b758-43e3-9259-0ef57bcd2b6a/serve-healthcheck-canary/0.log" Jul 06 19:28:21.209243 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:21.209208 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:21.209691 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:21.209298 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:21.209691 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:21.209351 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:21.209691 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:21.209359 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:21.209691 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:21.209411 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:25.209387856 +0000 UTC m=+146.084934515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:21.209691 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:21.209470 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:25.20945027 +0000 UTC m=+146.084996943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:25.016812 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:25.016783 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:25.140135 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:25.140107 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:25.242868 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:25.242842 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:25.243028 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:25.242913 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:25.243028 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:25.242978 2545 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:28:25.243028 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:25.242987 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:25.243028 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:25.243023 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:33.243010919 +0000 UTC m=+154.118557577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : configmap "kube-rbac-proxy" not found Jul 06 19:28:25.243182 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:25.243037 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:33.243030612 +0000 UTC m=+154.118577271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:25.998579 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:25.998553 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:26.116297 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:26.116261 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:26.537663 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:26.537631 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:26.637396 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:26.637368 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:27.222692 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:27.222658 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:27.323181 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:27.323150 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:28.804249 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:28.804220 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:28.932032 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:28.931994 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:29.745862 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:29.745833 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:30.008617 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:30.008544 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:32.880806 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:32.880774 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:32.994354 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:32.994322 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:33.096198 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:33.096170 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:33.209030 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:33.209005 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:33.303804 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:33.303779 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:33.303965 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:33.303819 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:33.303965 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:33.303905 2545 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:28:33.303965 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:33.303953 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls podName:5e62983e-ae49-42d3-b0c1-293fee95364f nodeName:}" failed. No retries permitted until 2026-07-06 19:28:49.303939665 +0000 UTC m=+170.179486337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nzb5n" (UID: "5e62983e-ae49-42d3-b0c1-293fee95364f") : secret "insights-runtime-extractor-tls" not found Jul 06 19:28:33.304351 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:33.304329 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e62983e-ae49-42d3-b0c1-293fee95364f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:33.783481 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:33.783450 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:34.176501 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:34.176465 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:34.400077 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:34.400042 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:34.773555 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:34.773522 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:35.172530 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:35.172499 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:35.581695 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:35.581617 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:35.776281 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:35.776232 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:35.931288 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:35.931264 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:36.239661 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:36.239582 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:36.798347 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:36.798315 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:36.978983 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:36.978955 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:37.393114 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:37.393084 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:37.631718 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:37.631691 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:37.804259 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:37.804187 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:38.065132 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:38.065051 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" podUID="2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b" Jul 06 19:28:38.079024 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:38.078997 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:38.088093 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:38.088063 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-8545644b6-jngbb" podUID="c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" Jul 06 19:28:38.095205 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:38.095179 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-sqdtt" podUID="9b877b8c-c234-4d29-aa5f-078c5de3726e" Jul 06 19:28:38.207905 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:38.207855 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:38.279279 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:38.279259 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:28:38.279389 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:38.279287 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqdtt" Jul 06 19:28:38.279483 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:38.279469 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:28:38.614323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:38.614291 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:39.167189 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:39.167161 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:39.350597 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:39.350572 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:39.638491 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:28:39.638466 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sv6xg" podUID="ed68dfa2-ff3c-4f7c-bf88-414559a5c413" Jul 06 19:28:39.759552 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:39.759514 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:40.060427 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:40.060393 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:40.207969 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:40.207892 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:40.860609 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:40.860580 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.192168 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:41.192138 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.430432 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:41.430381 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:41.964676 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:41.964644 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.229729 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:42.229651 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.379927 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:42.379901 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.468552 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:42.468520 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.601304 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:42.601234 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.777923 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:42.777893 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:42.880106 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:42.880041 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:43.077949 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.077916 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:28:43.078119 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.077970 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:28:43.078119 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.078025 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:28:43.080518 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.080491 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b877b8c-c234-4d29-aa5f-078c5de3726e-metrics-tls\") pod \"dns-default-sqdtt\" (UID: \"9b877b8c-c234-4d29-aa5f-078c5de3726e\") " pod="openshift-dns/dns-default-sqdtt" Jul 06 19:28:43.080653 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.080539 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"image-registry-8545644b6-jngbb\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:28:43.080653 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.080545 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4gj67\" (UID: \"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:28:43.184542 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.184513 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:43.376516 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.376485 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:43.383349 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.383329 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-s9hqf\"" Jul 06 19:28:43.383349 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.383340 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cvbvc\"" Jul 06 19:28:43.383513 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.383372 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7f4qs\"" Jul 06 19:28:43.390857 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.390832 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" Jul 06 19:28:43.390857 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.390852 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqdtt" Jul 06 19:28:43.390999 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.390895 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:28:43.548559 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.548533 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqdtt"] Jul 06 19:28:43.551487 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:28:43.551457 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b877b8c_c234_4d29_aa5f_078c5de3726e.slice/crio-9cec097cba6600dffd7b62d762ff489a0ea70f0d05e2ee6a59e4759f307737ab WatchSource:0}: Error finding container 9cec097cba6600dffd7b62d762ff489a0ea70f0d05e2ee6a59e4759f307737ab: Status 404 returned error can't find the container with id 9cec097cba6600dffd7b62d762ff489a0ea70f0d05e2ee6a59e4759f307737ab Jul 06 19:28:43.582468 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.582443 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:43.677638 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.677608 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:43.769764 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.769622 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-4gj67"] Jul 06 19:28:43.773152 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:28:43.773125 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb6c8ee_b9f3_485f_bf40_e4457aec4f1b.slice/crio-0c2d296bd5ef83c3089790d4e2d4e9c50b7d1d9f0deb540540b90d431222d7ab WatchSource:0}: Error finding container 0c2d296bd5ef83c3089790d4e2d4e9c50b7d1d9f0deb540540b90d431222d7ab: Status 404 returned error can't find the container with id 0c2d296bd5ef83c3089790d4e2d4e9c50b7d1d9f0deb540540b90d431222d7ab Jul 06 19:28:43.776000 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:28:43.775969 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d2a5c9_041a_4ea3_938f_a57b7a2c3bd2.slice/crio-f34347ce71aca79f102def7ad419d7ab75596321b5263721be875721504e1d19 WatchSource:0}: Error finding container f34347ce71aca79f102def7ad419d7ab75596321b5263721be875721504e1d19: Status 404 returned error can't find the container with id f34347ce71aca79f102def7ad419d7ab75596321b5263721be875721504e1d19 Jul 06 19:28:43.777835 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.777814 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8545644b6-jngbb"] Jul 06 19:28:43.818247 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.818222 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:43.992439 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:43.992399 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:44.091549 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.091469 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:44.301443 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.301011 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8545644b6-jngbb" event={"ID":"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2","Type":"ContainerStarted","Data":"e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee"} Jul 06 19:28:44.301443 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.301072 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8545644b6-jngbb" event={"ID":"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2","Type":"ContainerStarted","Data":"f34347ce71aca79f102def7ad419d7ab75596321b5263721be875721504e1d19"} Jul 06 19:28:44.301443 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.301134 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:28:44.302758 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.302727 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqdtt" event={"ID":"9b877b8c-c234-4d29-aa5f-078c5de3726e","Type":"ContainerStarted","Data":"9cec097cba6600dffd7b62d762ff489a0ea70f0d05e2ee6a59e4759f307737ab"} Jul 06 19:28:44.304647 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.304460 2545 generic.go:356] "Generic (PLEG): container finished" podID="95ca4ab3-d09e-4019-9a9c-7d31dfd509ab" containerID="c11aabedee5f0e1a64fa499cf46f8d649f3bf5f2e151490b8138eb057bcd045c" exitCode=1 Jul 06 19:28:44.304647 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.304533 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" event={"ID":"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab","Type":"ContainerDied","Data":"c11aabedee5f0e1a64fa499cf46f8d649f3bf5f2e151490b8138eb057bcd045c"} Jul 06 19:28:44.305128 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.304856 2545 scope.go:117] "RemoveContainer" containerID="c11aabedee5f0e1a64fa499cf46f8d649f3bf5f2e151490b8138eb057bcd045c" Jul 06 19:28:44.306016 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.305994 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" event={"ID":"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b","Type":"ContainerStarted","Data":"0c2d296bd5ef83c3089790d4e2d4e9c50b7d1d9f0deb540540b90d431222d7ab"} Jul 06 19:28:44.350253 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.350161 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8545644b6-jngbb" podStartSLOduration=164.350140134 podStartE2EDuration="2m44.350140134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:44.330535529 +0000 UTC m=+165.206082211" watchObservedRunningTime="2026-07-06 19:28:44.350140134 +0000 UTC m=+165.225686796" Jul 06 19:28:44.935011 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:44.934978 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:28:45.311825 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:45.311792 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" event={"ID":"95ca4ab3-d09e-4019-9a9c-7d31dfd509ab","Type":"ContainerStarted","Data":"8e8ee8923175103453eb8d2efd5223fe212b711fbcc2da67c76cf7c880957e9c"} Jul 06 19:28:45.312191 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:45.312133 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:28:45.313024 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:45.313005 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-546866d9f-jgcqg" Jul 06 19:28:45.313551 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:45.313527 2545 generic.go:356] "Generic (PLEG): container finished" podID="c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6" containerID="f30d263a4bfd43e5629c2271bcf04fec23fbbfa44fb9dc89a1c09bcf34eb39f2" exitCode=255 Jul 06 19:28:45.313712 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:45.313684 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" event={"ID":"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6","Type":"ContainerDied","Data":"f30d263a4bfd43e5629c2271bcf04fec23fbbfa44fb9dc89a1c09bcf34eb39f2"} Jul 06 19:28:45.313964 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:45.313949 2545 scope.go:117] "RemoveContainer" containerID="f30d263a4bfd43e5629c2271bcf04fec23fbbfa44fb9dc89a1c09bcf34eb39f2" Jul 06 19:28:45.373018 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:45.372945 2545 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" Jul 06 19:28:46.318332 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:46.318294 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqdtt" event={"ID":"9b877b8c-c234-4d29-aa5f-078c5de3726e","Type":"ContainerStarted","Data":"0b61af05a813e64510bb4137a06b2dbd8defdfaca444f548bfb39c63ca06d7b2"} Jul 06 19:28:46.318805 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:46.318353 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqdtt" event={"ID":"9b877b8c-c234-4d29-aa5f-078c5de3726e","Type":"ContainerStarted","Data":"79227e123790e508a8b48ce71d8d56022bee97fc52463915894eea56ae393091"} Jul 06 19:28:46.318805 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:46.318379 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sqdtt" Jul 06 19:28:46.319806 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:46.319781 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" event={"ID":"2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b","Type":"ContainerStarted","Data":"ede4bf57cad07ad1c81b12a44c08acbcb54465b82d3ed1696cc5e5be73331d6d"} Jul 06 19:28:46.321455 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:46.321436 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-859566fb64-mjhsw" event={"ID":"c97abb00-0cfe-4e3a-a181-e0da7bcb3ce6","Type":"ContainerStarted","Data":"1bba2b8930b1472a7e4f558d46f6048302224dcbe3e6874c40ec2f48cb1f807d"} Jul 06 19:28:46.357741 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:46.357700 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sqdtt" podStartSLOduration=129.647909278 podStartE2EDuration="2m11.357688107s" totalImagesPullingTime="1.709778829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:46.355915846 +0000 UTC m=+167.231462549" watchObservedRunningTime="2026-07-06 19:28:46.357688107 +0000 UTC m=+167.233234788" Jul 06 19:28:46.387716 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:46.387675 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4gj67" podStartSLOduration=159.903896299 podStartE2EDuration="2m41.387661498s" totalImagesPullingTime="1.483765199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:46.387489803 +0000 UTC m=+167.263036482" watchObservedRunningTime="2026-07-06 19:28:46.387661498 +0000 UTC m=+167.263208178" Jul 06 19:28:47.146253 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:47.146218 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:47.570760 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:47.570670 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:47.824476 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:47.824384 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:47.936044 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:47.936011 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:49.319328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:49.319294 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:49.321609 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:49.321588 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e62983e-ae49-42d3-b0c1-293fee95364f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nzb5n\" (UID: \"5e62983e-ae49-42d3-b0c1-293fee95364f\") " pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:49.486776 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:49.486739 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nzb5n" Jul 06 19:28:49.612908 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:49.612833 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nzb5n"] Jul 06 19:28:49.615820 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:28:49.615795 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e62983e_ae49_42d3_b0c1_293fee95364f.slice/crio-3b3ecb9e25b6d74d4b185a01bad22dee97f54662df6e25a8dd7106973a2c1b03 WatchSource:0}: Error finding container 3b3ecb9e25b6d74d4b185a01bad22dee97f54662df6e25a8dd7106973a2c1b03: Status 404 returned error can't find the container with id 3b3ecb9e25b6d74d4b185a01bad22dee97f54662df6e25a8dd7106973a2c1b03 Jul 06 19:28:49.658915 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:49.658881 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:49.782951 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:49.782919 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:50.337831 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:50.337792 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nzb5n" event={"ID":"5e62983e-ae49-42d3-b0c1-293fee95364f","Type":"ContainerStarted","Data":"3b3ecb9e25b6d74d4b185a01bad22dee97f54662df6e25a8dd7106973a2c1b03"} Jul 06 19:28:50.820940 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:50.820908 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:50.936870 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:50.936829 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:52.114216 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:52.114184 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:52.247200 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:52.247171 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:53.122335 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:53.122284 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:53.301898 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:53.301864 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:53.462776 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:53.462742 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:53.598887 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:53.598855 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:28:54.354833 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:54.354757 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nzb5n" event={"ID":"5e62983e-ae49-42d3-b0c1-293fee95364f","Type":"ContainerStarted","Data":"31796e306254b39a8d6d00545da4cdc31124b613d3c6646e78d92c942aab4dd8"} Jul 06 19:28:54.354833 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:54.354832 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nzb5n" event={"ID":"5e62983e-ae49-42d3-b0c1-293fee95364f","Type":"ContainerStarted","Data":"b727847bd5e0dafe1c5d062d1b4f80506efff37acf97cfed5fa65df74267b25e"} Jul 06 19:28:56.328263 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:56.328233 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sqdtt" Jul 06 19:28:56.523199 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:56.523172 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:56.664691 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:56.664663 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:57.293479 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:57.293439 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:57.369078 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:57.369040 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nzb5n" event={"ID":"5e62983e-ae49-42d3-b0c1-293fee95364f","Type":"ContainerStarted","Data":"5f08b80048029fb8ede6440dd6c0ef25ad8f5370f5758a651cf2e916d56b4dc4"} Jul 06 19:28:57.402066 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:57.402019 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nzb5n" podStartSLOduration=34.630421753 podStartE2EDuration="41.402004233s" totalImagesPullingTime="6.77158248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:28:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:57.401710868 +0000 UTC m=+178.277257550" watchObservedRunningTime="2026-07-06 19:28:57.402004233 +0000 UTC m=+178.277550919" Jul 06 19:28:58.651742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:58.651713 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:58.768125 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:58.768093 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:28:59.179641 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:28:59.179613 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:00.717459 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:00.717410 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:01.121314 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.121233 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:01.229145 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.229118 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tlgv9"] Jul 06 19:29:01.232512 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.232498 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.234780 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.234760 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:29:01.235071 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.235037 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7b468\"" Jul 06 19:29:01.235352 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.235331 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:29:01.235462 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.235398 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:29:01.235645 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.235628 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:29:01.235721 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.235635 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:29:01.235897 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.235883 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:29:01.308470 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308438 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-textfile\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.308470 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308470 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-wtmp\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.308674 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308487 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-tls\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.308674 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308512 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-root\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.308674 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308558 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-sys\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.308674 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308616 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/063be4ab-50c4-4a34-9065-6e09067cefe4-metrics-client-ca\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.308674 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308638 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.308674 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308667 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.308867 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.308708 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmz8\" (UniqueName: \"kubernetes.io/projected/063be4ab-50c4-4a34-9065-6e09067cefe4-kube-api-access-2dmz8\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409223 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409197 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-textfile\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409387 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409229 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-wtmp\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409387 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409271 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-tls\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409387 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409303 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-root\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409387 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409325 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-sys\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409387 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409363 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/063be4ab-50c4-4a34-9065-6e09067cefe4-metrics-client-ca\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409393 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409409 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-root\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409407 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-wtmp\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409459 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/063be4ab-50c4-4a34-9065-6e09067cefe4-sys\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:29:01.409479 2545 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409465 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409540 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmz8\" (UniqueName: \"kubernetes.io/projected/063be4ab-50c4-4a34-9065-6e09067cefe4-kube-api-access-2dmz8\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:29:01.409620 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-tls podName:063be4ab-50c4-4a34-9065-6e09067cefe4 nodeName:}" failed. No retries permitted until 2026-07-06 19:29:01.909599219 +0000 UTC m=+182.785145878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-tls") pod "node-exporter-tlgv9" (UID: "063be4ab-50c4-4a34-9065-6e09067cefe4") : secret "node-exporter-tls" not found Jul 06 19:29:01.409736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409636 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-textfile\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.410149 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.409897 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-accelerators-collector-config\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.410149 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.410045 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/063be4ab-50c4-4a34-9065-6e09067cefe4-metrics-client-ca\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.411791 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.411763 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.420594 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.420564 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:01.425625 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.425607 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmz8\" (UniqueName: \"kubernetes.io/projected/063be4ab-50c4-4a34-9065-6e09067cefe4-kube-api-access-2dmz8\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.610899 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.610867 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:01.751845 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.751767 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:01.913235 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.913204 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-tls\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.915642 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.915620 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/063be4ab-50c4-4a34-9065-6e09067cefe4-node-exporter-tls\") pod \"node-exporter-tlgv9\" (UID: \"063be4ab-50c4-4a34-9065-6e09067cefe4\") " pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:01.931977 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:01.931949 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:02.045166 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:02.045065 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:02.141991 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:02.141956 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tlgv9" Jul 06 19:29:02.153974 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:29:02.153951 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063be4ab_50c4_4a34_9065_6e09067cefe4.slice/crio-9d362e9c5aad4359309ea0031398816a636484485a4a95828a2b72958dce5783 WatchSource:0}: Error finding container 9d362e9c5aad4359309ea0031398816a636484485a4a95828a2b72958dce5783: Status 404 returned error can't find the container with id 9d362e9c5aad4359309ea0031398816a636484485a4a95828a2b72958dce5783 Jul 06 19:29:02.244131 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:02.244096 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:02.385657 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:02.385631 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlgv9" event={"ID":"063be4ab-50c4-4a34-9065-6e09067cefe4","Type":"ContainerStarted","Data":"9d362e9c5aad4359309ea0031398816a636484485a4a95828a2b72958dce5783"} Jul 06 19:29:02.569884 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:02.569851 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:02.789713 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:02.789633 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:03.390765 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:03.390733 2545 generic.go:356] "Generic (PLEG): container finished" podID="063be4ab-50c4-4a34-9065-6e09067cefe4" containerID="f263f050092b33cec874b4de4b07be27e6095deba4edc34948ef26cd827a1d92" exitCode=0 Jul 06 19:29:03.390947 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:03.390808 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlgv9" event={"ID":"063be4ab-50c4-4a34-9065-6e09067cefe4","Type":"ContainerDied","Data":"f263f050092b33cec874b4de4b07be27e6095deba4edc34948ef26cd827a1d92"} Jul 06 19:29:03.395049 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:03.395023 2545 patch_prober.go:28] interesting pod/image-registry-8545644b6-jngbb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 19:29:03.395146 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:03.395080 2545 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8545644b6-jngbb" podUID="c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:29:04.263879 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:04.263834 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:04.366651 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:04.366621 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:04.396265 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:04.396234 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlgv9" event={"ID":"063be4ab-50c4-4a34-9065-6e09067cefe4","Type":"ContainerStarted","Data":"ead134baadbd43fa1504f89542804b8765c58d828e77ab7bc8c043a07742019f"} Jul 06 19:29:04.396396 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:04.396279 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tlgv9" event={"ID":"063be4ab-50c4-4a34-9065-6e09067cefe4","Type":"ContainerStarted","Data":"011fb8cd11011f58b936b378f6feb2b8c295e4cc1b6cad261ccfa4f0074ed473"} Jul 06 19:29:04.421974 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:04.421932 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tlgv9" podStartSLOduration=2.352032078 podStartE2EDuration="3.421916093s" totalImagesPullingTime="997.66925ms" totalInitContainerRuntime="72.214765ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:29:04.420620221 +0000 UTC m=+185.296166901" watchObservedRunningTime="2026-07-06 19:29:04.421916093 +0000 UTC m=+185.297462773" Jul 06 19:29:04.485326 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:04.485294 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:04.683323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:04.683296 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:05.318192 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:05.318165 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:29:05.521229 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:05.521196 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:05.622118 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:05.622046 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:05.743533 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:05.743497 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:05.844956 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:05.844926 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:05.970044 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:05.970016 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:07.622731 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:07.622696 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:08.863955 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:08.863924 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:09.010119 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:09.010089 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:12.482939 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:12.482907 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:12.579171 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:12.579143 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:13.252662 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:13.252624 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:13.409938 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:13.409907 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:14.651003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:14.650968 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:14.768674 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:14.768647 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:14.832840 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:14.832810 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8545644b6-jngbb"] Jul 06 19:29:14.981330 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:14.981257 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:15.086736 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:15.086706 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:15.358177 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:15.358093 2545 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" podUID="4c720a33-92fe-4330-8d54-f2ed209f785d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:29:17.373378 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:17.373347 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:17.545215 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:17.545186 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:17.751645 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:17.751620 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:17.851981 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:17.851949 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:17.954135 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:17.954109 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:20.847668 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:20.847635 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:20.943649 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:20.943622 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:21.589141 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:21.589105 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:21.686533 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:21.686500 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:22.349451 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:22.349394 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:22.450298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:22.450262 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:25.358109 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:25.358076 2545 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" podUID="4c720a33-92fe-4330-8d54-f2ed209f785d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:29:29.072784 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:29.072757 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:29.181060 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:29.181032 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:29.276918 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:29.276866 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:32.745003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:32.744971 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:32.845710 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:32.845683 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:32.950561 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:32.950534 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:35.357631 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:35.357599 2545 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" podUID="4c720a33-92fe-4330-8d54-f2ed209f785d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 06 19:29:35.358047 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:35.357657 2545 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" Jul 06 19:29:35.358129 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:35.358099 2545 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"23da1ee4dd5a0000cfe42e370e1222f68819634a86785d7805d7f95f36216d52"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 06 19:29:35.358163 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:35.358150 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" podUID="4c720a33-92fe-4330-8d54-f2ed209f785d" containerName="service-proxy" containerID="cri-o://23da1ee4dd5a0000cfe42e370e1222f68819634a86785d7805d7f95f36216d52" gracePeriod=30 Jul 06 19:29:35.492988 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:35.492962 2545 generic.go:356] "Generic (PLEG): container finished" podID="4c720a33-92fe-4330-8d54-f2ed209f785d" containerID="23da1ee4dd5a0000cfe42e370e1222f68819634a86785d7805d7f95f36216d52" exitCode=2 Jul 06 19:29:35.493107 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:35.493025 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" event={"ID":"4c720a33-92fe-4330-8d54-f2ed209f785d","Type":"ContainerDied","Data":"23da1ee4dd5a0000cfe42e370e1222f68819634a86785d7805d7f95f36216d52"} Jul 06 19:29:35.834193 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:35.834160 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:35.924432 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:35.924376 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:36.498278 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:36.498233 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cd8c5f557-fvbct" event={"ID":"4c720a33-92fe-4330-8d54-f2ed209f785d","Type":"ContainerStarted","Data":"879169fb1d49ec3f89f846e8f0b57a00a71842b4d54b0a824ae9138f1c62b714"} Jul 06 19:29:37.673256 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:37.673217 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:37.753185 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:37.753156 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:39.855174 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:39.855131 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8545644b6-jngbb" podUID="c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" containerName="registry" containerID="cri-o://e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee" gracePeriod=30 Jul 06 19:29:40.098544 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.098517 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:29:40.190323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190294 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-trusted-ca\") pod \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " Jul 06 19:29:40.190323 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190326 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-bound-sa-token\") pod \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " Jul 06 19:29:40.190569 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190342 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") pod \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " Jul 06 19:29:40.190569 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190362 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-image-registry-private-configuration\") pod \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " Jul 06 19:29:40.190569 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190382 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-certificates\") pod \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " Jul 06 19:29:40.190569 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190441 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-installation-pull-secrets\") pod \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " Jul 06 19:29:40.190569 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190466 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xc7b\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-kube-api-access-5xc7b\") pod \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " Jul 06 19:29:40.190569 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190498 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-ca-trust-extracted\") pod \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\" (UID: \"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2\") " Jul 06 19:29:40.190872 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190716 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:40.190872 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.190813 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:29:40.192788 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.192766 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:40.192898 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.192823 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:40.193144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.193089 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:29:40.193369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.193252 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:29:40.193369 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.193349 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-kube-api-access-5xc7b" (OuterVolumeSpecName: "kube-api-access-5xc7b") pod "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2"). InnerVolumeSpecName "kube-api-access-5xc7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:29:40.199502 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.199482 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" (UID: "c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:29:40.291734 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.291713 2545 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-trusted-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:29:40.291734 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.291733 2545 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-bound-sa-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:29:40.291841 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.291741 2545 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:29:40.291841 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.291749 2545 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-image-registry-private-configuration\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:29:40.291841 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.291757 2545 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-registry-certificates\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:29:40.291841 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.291766 2545 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-installation-pull-secrets\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:29:40.291841 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.291773 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xc7b\" (UniqueName: \"kubernetes.io/projected/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-kube-api-access-5xc7b\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:29:40.291841 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.291781 2545 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2-ca-trust-extracted\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:29:40.431831 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.431803 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:40.512872 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.512795 2545 generic.go:356] "Generic (PLEG): container finished" podID="c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" containerID="e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee" exitCode=0 Jul 06 19:29:40.512872 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.512851 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8545644b6-jngbb" Jul 06 19:29:40.512872 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.512865 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8545644b6-jngbb" event={"ID":"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2","Type":"ContainerDied","Data":"e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee"} Jul 06 19:29:40.513052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.512934 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8545644b6-jngbb" event={"ID":"c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2","Type":"ContainerDied","Data":"f34347ce71aca79f102def7ad419d7ab75596321b5263721be875721504e1d19"} Jul 06 19:29:40.513052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.512955 2545 scope.go:117] "RemoveContainer" containerID="e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee" Jul 06 19:29:40.521240 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.521225 2545 scope.go:117] "RemoveContainer" containerID="e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee" Jul 06 19:29:40.521534 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:29:40.521514 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee\": container with ID starting with e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee not found: ID does not exist" containerID="e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee" Jul 06 19:29:40.521614 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.521543 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee"} err="failed to get container status \"e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee\": rpc error: code = NotFound desc = could not find container \"e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee\": container with ID starting with e9d786ed9c2cc10f15d88b0df3cfa5a748c421c0670366be236ec3bc4ebd0bee not found: ID does not exist" Jul 06 19:29:40.537633 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.537609 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8545644b6-jngbb"] Jul 06 19:29:40.550231 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:40.550211 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8545644b6-jngbb"] Jul 06 19:29:41.602221 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:41.602188 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" path="/var/lib/kubelet/pods/c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2/volumes" Jul 06 19:29:54.440470 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:54.440434 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:29:59.458648 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:29:59.458615 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:10.285799 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:10.285765 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:11.308185 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:11.308152 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:11.521023 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:11.520973 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:30:11.523398 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:11.523378 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed68dfa2-ff3c-4f7c-bf88-414559a5c413-metrics-certs\") pod \"network-metrics-daemon-sv6xg\" (UID: \"ed68dfa2-ff3c-4f7c-bf88-414559a5c413\") " pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:30:11.601848 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:11.601764 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5qdm\"" Jul 06 19:30:11.610120 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:11.610104 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sv6xg" Jul 06 19:30:11.739757 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:11.739728 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sv6xg"] Jul 06 19:30:11.743547 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:30:11.743523 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded68dfa2_ff3c_4f7c_bf88_414559a5c413.slice/crio-b4335f2fe23f30691c54f7df0a12c44082ebb14baef42f89c1a92990207c5dee WatchSource:0}: Error finding container b4335f2fe23f30691c54f7df0a12c44082ebb14baef42f89c1a92990207c5dee: Status 404 returned error can't find the container with id b4335f2fe23f30691c54f7df0a12c44082ebb14baef42f89c1a92990207c5dee Jul 06 19:30:12.616782 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:12.616745 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sv6xg" event={"ID":"ed68dfa2-ff3c-4f7c-bf88-414559a5c413","Type":"ContainerStarted","Data":"b4335f2fe23f30691c54f7df0a12c44082ebb14baef42f89c1a92990207c5dee"} Jul 06 19:30:13.621775 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:13.621733 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sv6xg" event={"ID":"ed68dfa2-ff3c-4f7c-bf88-414559a5c413","Type":"ContainerStarted","Data":"d14e9c935f30e62eb24c5391cc4712310c07ad675eb414b4e3c11b183d917efb"} Jul 06 19:30:13.622163 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:13.621803 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sv6xg" event={"ID":"ed68dfa2-ff3c-4f7c-bf88-414559a5c413","Type":"ContainerStarted","Data":"0d899a558a7c6d57f27b9a77c98c22c884d9c1a9bd1d93f2ef04f573675d9638"} Jul 06 19:30:13.640677 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:13.640627 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sv6xg" podStartSLOduration=252.670632499 podStartE2EDuration="4m13.640610812s" totalImagesPullingTime="969.978313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:13.639290753 +0000 UTC m=+254.514837433" watchObservedRunningTime="2026-07-06 19:30:13.640610812 +0000 UTC m=+254.516157528" Jul 06 19:30:13.791430 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:13.791373 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:13.893062 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:13.893027 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:17.462977 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:17.462947 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:17.565208 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:17.565181 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:23.464653 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:23.464569 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:23.592322 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:23.592288 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:29.533572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:29.533542 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:29.622021 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:29.621993 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:35.389353 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:35.389321 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:35.492682 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:35.492649 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:35.579712 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:35.579687 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:35.855760 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:35.855675 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:35.948205 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:35.948170 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:36.127192 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:36.127120 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:36.223037 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:36.223010 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:46.099744 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:46.099717 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:46.214802 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:46.214772 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:30:59.504106 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:59.504075 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:30:59.511358 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:59.511338 2545 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:30:59.511514 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:30:59.511376 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:31:14.745205 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:14.745173 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:14.892883 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:14.892851 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:15.942793 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:15.942750 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:16.100175 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:16.100146 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:16.854043 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:16.854014 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:17.122481 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:17.122389 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:43.192830 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:43.192802 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:44.128342 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:44.128312 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:47.295437 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:47.295386 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:47.400496 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:47.400461 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:48.536525 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:48.536496 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:48.637594 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:48.637562 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:49.562306 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:49.562277 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:49.707281 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:49.707254 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:52.171865 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:52.171786 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:52.425245 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:52.425169 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:54.026991 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:54.026960 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:54.373609 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:54.373545 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:57.325799 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:57.325766 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:57.415879 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:57.415837 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:59.169462 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:59.169433 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:59.404678 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:59.404652 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:31:59.560138 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:31:59.560058 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:05.100977 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:05.100940 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:05.215091 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:05.215059 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:07.025036 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:07.024998 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:10.244722 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:10.244690 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:21.663373 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:21.663342 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:21.787511 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:21.787485 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:22.297508 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.297474 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:22.515926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.515897 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:22.626697 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.626624 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:22.634609 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.634587 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp"] Jul 06 19:32:22.634800 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.634788 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" containerName="registry" Jul 06 19:32:22.634856 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.634801 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" containerName="registry" Jul 06 19:32:22.634856 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.634852 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d2a5c9-041a-4ea3-938f-a57b7a2c3bd2" containerName="registry" Jul 06 19:32:22.637634 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.637618 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.640209 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.640167 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7kjrq\"" Jul 06 19:32:22.640209 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.640167 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:32:22.640388 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.640192 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:32:22.640594 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.640577 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:32:22.640664 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.640581 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:32:22.655218 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.655190 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp"] Jul 06 19:32:22.710589 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.710554 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.711012 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.710609 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.711012 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.710633 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-tmp\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.711012 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.710708 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-manifests\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.711012 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.710895 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8vbs\" (UniqueName: \"kubernetes.io/projected/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-kube-api-access-g8vbs\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.808895 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.808864 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:22.811245 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.811227 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-manifests\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.811351 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.811275 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vbs\" (UniqueName: \"kubernetes.io/projected/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-kube-api-access-g8vbs\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.811351 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.811312 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.811351 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.811342 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.811525 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.811368 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-tmp\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.811606 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.811582 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-manifests\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.813686 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.813665 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-tmp\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.813807 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.813792 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.813989 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.813966 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.820774 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.820751 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vbs\" (UniqueName: \"kubernetes.io/projected/57d7db8a-d2d4-45d3-923e-bd8dffb2faa3-kube-api-access-g8vbs\") pod \"opendatahub-operator-controller-manager-5586496c87-6gxfp\" (UID: \"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3\") " pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:22.914499 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.914467 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:22.947620 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:22.947584 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:23.020606 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:23.020579 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:23.102107 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:23.102078 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp"] Jul 06 19:32:23.105714 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:32:23.105689 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d7db8a_d2d4_45d3_923e_bd8dffb2faa3.slice/crio-bc60786d56363189944fdd8be5d41ec0b0c9b8de1b87b47739db66024f696deb WatchSource:0}: Error finding container bc60786d56363189944fdd8be5d41ec0b0c9b8de1b87b47739db66024f696deb: Status 404 returned error can't find the container with id bc60786d56363189944fdd8be5d41ec0b0c9b8de1b87b47739db66024f696deb Jul 06 19:32:23.107332 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:23.107317 2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:32:23.159939 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:23.159907 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:23.270956 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:23.270865 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:23.368247 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:23.368220 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:24.035183 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:24.035129 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" event={"ID":"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3","Type":"ContainerStarted","Data":"bc60786d56363189944fdd8be5d41ec0b0c9b8de1b87b47739db66024f696deb"} Jul 06 19:32:25.440371 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:25.440332 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:25.667342 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:25.667312 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:27.048209 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:27.048176 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" event={"ID":"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3","Type":"ContainerStarted","Data":"e9ed409ce74f58d083b2d731dec99c78286959dd2c42f3416c37611e6abc1e20"} Jul 06 19:32:28.052616 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:28.052584 2545 generic.go:356] "Generic (PLEG): container finished" podID="57d7db8a-d2d4-45d3-923e-bd8dffb2faa3" containerID="e9ed409ce74f58d083b2d731dec99c78286959dd2c42f3416c37611e6abc1e20" exitCode=0 Jul 06 19:32:28.052983 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:28.052667 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" event={"ID":"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3","Type":"ContainerDied","Data":"e9ed409ce74f58d083b2d731dec99c78286959dd2c42f3416c37611e6abc1e20"} Jul 06 19:32:29.058043 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:29.058013 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" event={"ID":"57d7db8a-d2d4-45d3-923e-bd8dffb2faa3","Type":"ContainerStarted","Data":"c8c3d3e8bdcf9a8634b0dbba6a58182edeefe44fc975a1e6fc68f0c48492bcc2"} Jul 06 19:32:29.058409 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:29.058125 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:29.091560 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:29.091515 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" podStartSLOduration=2.463582859 podStartE2EDuration="7.091502114s" totalImagesPullingTime="3.585831093s" totalInitContainerRuntime="1.042088162s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:29.091262983 +0000 UTC m=+389.966809665" watchObservedRunningTime="2026-07-06 19:32:29.091502114 +0000 UTC m=+389.967048785" Jul 06 19:32:39.815666 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.815634 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb"] Jul 06 19:32:39.824580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.824557 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:39.828024 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.827998 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:32:39.828024 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.828022 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:32:39.828212 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.828035 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:32:39.828212 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.828052 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:32:39.828212 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.827999 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-znr6f\"" Jul 06 19:32:39.828212 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.828166 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:32:39.833714 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.833694 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb"] Jul 06 19:32:39.922496 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.922472 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c092ad3-3008-4a3e-bf03-963b4d69388a-cert\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:39.922627 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.922507 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c092ad3-3008-4a3e-bf03-963b4d69388a-metrics-cert\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:39.922627 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.922578 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsrj\" (UniqueName: \"kubernetes.io/projected/6c092ad3-3008-4a3e-bf03-963b4d69388a-kube-api-access-gqsrj\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:39.922701 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:39.922665 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c092ad3-3008-4a3e-bf03-963b4d69388a-manager-config\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.023688 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.023663 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c092ad3-3008-4a3e-bf03-963b4d69388a-manager-config\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.023798 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.023700 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c092ad3-3008-4a3e-bf03-963b4d69388a-cert\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.023798 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.023722 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c092ad3-3008-4a3e-bf03-963b4d69388a-metrics-cert\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.023798 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.023752 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsrj\" (UniqueName: \"kubernetes.io/projected/6c092ad3-3008-4a3e-bf03-963b4d69388a-kube-api-access-gqsrj\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.024328 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.024307 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c092ad3-3008-4a3e-bf03-963b4d69388a-manager-config\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.026173 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.026153 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c092ad3-3008-4a3e-bf03-963b4d69388a-metrics-cert\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.026265 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.026212 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c092ad3-3008-4a3e-bf03-963b4d69388a-cert\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.034910 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.034888 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsrj\" (UniqueName: \"kubernetes.io/projected/6c092ad3-3008-4a3e-bf03-963b4d69388a-kube-api-access-gqsrj\") pod \"lws-controller-manager-94489db4c-b5dqb\" (UID: \"6c092ad3-3008-4a3e-bf03-963b4d69388a\") " pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.063446 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.063407 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5586496c87-6gxfp" Jul 06 19:32:40.133720 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.133651 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:40.260886 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:40.260854 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb"] Jul 06 19:32:40.263956 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:32:40.263925 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c092ad3_3008_4a3e_bf03_963b4d69388a.slice/crio-14f8650b1ee77c9cc20c4b365521cd8fc8caf2eeb40a48873d4b6c7a72cf3e24 WatchSource:0}: Error finding container 14f8650b1ee77c9cc20c4b365521cd8fc8caf2eeb40a48873d4b6c7a72cf3e24: Status 404 returned error can't find the container with id 14f8650b1ee77c9cc20c4b365521cd8fc8caf2eeb40a48873d4b6c7a72cf3e24 Jul 06 19:32:41.096961 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:41.096920 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" event={"ID":"6c092ad3-3008-4a3e-bf03-963b4d69388a","Type":"ContainerStarted","Data":"14f8650b1ee77c9cc20c4b365521cd8fc8caf2eeb40a48873d4b6c7a72cf3e24"} Jul 06 19:32:41.874904 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:41.874848 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:42.100209 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.100171 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:42.595202 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.595170 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:42.643848 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.643821 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl"] Jul 06 19:32:42.646688 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.646673 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.648972 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.648948 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:32:42.649218 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.649198 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:32:42.649294 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.649250 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-swnw8\"" Jul 06 19:32:42.649741 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.649721 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:32:42.652926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.652761 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:32:42.682892 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.682857 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl"] Jul 06 19:32:42.745065 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.744986 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-tmp\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.745065 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.745028 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-tls-certs\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.745252 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.745089 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spr8v\" (UniqueName: \"kubernetes.io/projected/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-kube-api-access-spr8v\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.845649 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.845616 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-tmp\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.845820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.845659 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-tls-certs\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.845820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.845698 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spr8v\" (UniqueName: \"kubernetes.io/projected/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-kube-api-access-spr8v\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.848083 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.848062 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-tmp\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.848276 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.848259 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-tls-certs\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.859751 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.859722 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spr8v\" (UniqueName: \"kubernetes.io/projected/7052a6a3-f7d2-4e06-a39e-2309f5ad66b8-kube-api-access-spr8v\") pod \"kube-auth-proxy-7c47f5985b-t46xl\" (UID: \"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8\") " pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.955848 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.955820 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" Jul 06 19:32:42.979246 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:42.979224 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:43.092210 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.092084 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl"] Jul 06 19:32:43.094706 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:32:43.094680 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7052a6a3_f7d2_4e06_a39e_2309f5ad66b8.slice/crio-f4bc1d16a38f14f661e1f448f95dc1fde3b1e99ae3bd7b1c95d9944b68012f4f WatchSource:0}: Error finding container f4bc1d16a38f14f661e1f448f95dc1fde3b1e99ae3bd7b1c95d9944b68012f4f: Status 404 returned error can't find the container with id f4bc1d16a38f14f661e1f448f95dc1fde3b1e99ae3bd7b1c95d9944b68012f4f Jul 06 19:32:43.105634 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.105609 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" event={"ID":"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8","Type":"ContainerStarted","Data":"f4bc1d16a38f14f661e1f448f95dc1fde3b1e99ae3bd7b1c95d9944b68012f4f"} Jul 06 19:32:43.106958 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.106932 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" event={"ID":"6c092ad3-3008-4a3e-bf03-963b4d69388a","Type":"ContainerStarted","Data":"65f10507d4191ade8b4b5b8b7b66cda8f9dd6e244cf6c724c7c858a4b2841444"} Jul 06 19:32:43.107062 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.107002 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:43.149499 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.149445 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" podStartSLOduration=1.971025147 podStartE2EDuration="4.149409269s" totalImagesPullingTime="2.178384122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:43.147772039 +0000 UTC m=+404.023318719" watchObservedRunningTime="2026-07-06 19:32:43.149409269 +0000 UTC m=+404.024955952" Jul 06 19:32:43.159560 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.159535 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:43.274625 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.274550 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:43.536924 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.536833 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:43.655458 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:43.655404 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:44.449579 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:44.449532 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:44.623476 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:44.623440 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:44.807945 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:44.807865 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:45.463830 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:45.463801 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:46.120446 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:46.120345 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" event={"ID":"7052a6a3-f7d2-4e06-a39e-2309f5ad66b8","Type":"ContainerStarted","Data":"ae5ace0b12e9f5f52dd096c4573f111c38d9cf819a959bbf9a6853e6d3cbb3fb"} Jul 06 19:32:46.140558 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:46.140497 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c47f5985b-t46xl" podStartSLOduration=1.5740767610000002 podStartE2EDuration="4.140479903s" totalImagesPullingTime="2.566403142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:32:46.138629685 +0000 UTC m=+407.014176366" watchObservedRunningTime="2026-07-06 19:32:46.140479903 +0000 UTC m=+407.016026585" Jul 06 19:32:46.231572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:46.231542 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:49.501238 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:49.501194 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:49.599174 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:49.599138 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:50.434569 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:50.434536 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:50.693759 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:50.693676 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:50.882379 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:50.882329 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:54.114002 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:54.113972 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:32:55.186760 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:55.186730 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:55.395588 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:55.395559 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:55.668371 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:55.668345 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:55.824464 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:55.824435 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:32:56.054694 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:32:56.054598 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:40.521995 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:33:40.521960 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:41.723912 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:33:41.723880 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:44.317840 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:33:44.317794 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:44.418532 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:33:44.418505 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:44.544496 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:33:44.544469 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:45.287479 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:33:45.287447 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:33:45.462295 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:33:45.462269 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:07.852551 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:07.852511 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:11.900515 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:11.900464 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:14.662456 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:14.662405 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:18.121895 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:18.121843 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:19.255727 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:19.255696 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:19.587247 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:19.587170 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:19.930986 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:19.930956 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:21.457694 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:21.457661 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:25.527930 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:25.527887 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:25.832340 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:25.832258 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:26.869592 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.869562 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt"] Jul 06 19:34:26.871717 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.871702 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:26.874138 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.874109 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 19:34:26.874279 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.874208 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:34:26.874345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.874276 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-62dw5\"" Jul 06 19:34:26.874345 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.874328 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 19:34:26.874741 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.874728 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:34:26.882644 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.882623 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt"] Jul 06 19:34:26.911091 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.911063 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:26.968525 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.968501 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8037be85-5de3-46eb-855c-06745c4dc394-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:26.968608 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.968536 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8037be85-5de3-46eb-855c-06745c4dc394-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:26.968608 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:26.968562 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzzc\" (UniqueName: \"kubernetes.io/projected/8037be85-5de3-46eb-855c-06745c4dc394-kube-api-access-tmzzc\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.068907 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.068882 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8037be85-5de3-46eb-855c-06745c4dc394-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.069050 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.068910 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzzc\" (UniqueName: \"kubernetes.io/projected/8037be85-5de3-46eb-855c-06745c4dc394-kube-api-access-tmzzc\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.069183 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.069164 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8037be85-5de3-46eb-855c-06745c4dc394-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.069285 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:34:27.069269 2545 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 19:34:27.069345 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:34:27.069325 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8037be85-5de3-46eb-855c-06745c4dc394-plugin-serving-cert podName:8037be85-5de3-46eb-855c-06745c4dc394 nodeName:}" failed. No retries permitted until 2026-07-06 19:34:27.569311293 +0000 UTC m=+508.444857952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/8037be85-5de3-46eb-855c-06745c4dc394-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-gcftt" (UID: "8037be85-5de3-46eb-855c-06745c4dc394") : secret "plugin-serving-cert" not found Jul 06 19:34:27.069609 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.069589 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8037be85-5de3-46eb-855c-06745c4dc394-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.084216 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.084196 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzzc\" (UniqueName: \"kubernetes.io/projected/8037be85-5de3-46eb-855c-06745c4dc394-kube-api-access-tmzzc\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.302041 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.302010 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:27.573558 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.573482 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8037be85-5de3-46eb-855c-06745c4dc394-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.576094 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.576069 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8037be85-5de3-46eb-855c-06745c4dc394-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gcftt\" (UID: \"8037be85-5de3-46eb-855c-06745c4dc394\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.780649 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.780609 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" Jul 06 19:34:27.909222 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:27.909191 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt"] Jul 06 19:34:27.912572 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:34:27.912545 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8037be85_5de3_46eb_855c_06745c4dc394.slice/crio-56d78f04e7a09a06e1237784d72586111e1347c4e4ee2252162f3861dd3dc64e WatchSource:0}: Error finding container 56d78f04e7a09a06e1237784d72586111e1347c4e4ee2252162f3861dd3dc64e: Status 404 returned error can't find the container with id 56d78f04e7a09a06e1237784d72586111e1347c4e4ee2252162f3861dd3dc64e Jul 06 19:34:28.473806 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:28.473767 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" event={"ID":"8037be85-5de3-46eb-855c-06745c4dc394","Type":"ContainerStarted","Data":"56d78f04e7a09a06e1237784d72586111e1347c4e4ee2252162f3861dd3dc64e"} Jul 06 19:34:29.076585 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:29.076550 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:29.191647 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:29.191613 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:32.677764 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:32.677733 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:32.793382 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:32.793345 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:40.705600 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:40.705560 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:40.816495 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:40.816464 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:41.920620 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:41.920586 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:42.038796 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:42.038764 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:43.347949 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:43.347910 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:43.451360 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:43.451328 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:46.246437 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:46.246385 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:46.408544 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:46.408502 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:49.754208 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:49.754164 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:49.860389 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:49.860354 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:51.566484 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:51.566439 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" event={"ID":"8037be85-5de3-46eb-855c-06745c4dc394","Type":"ContainerStarted","Data":"5c0867b4bb22ce7c8279c266e5e8d156771cec4b593cf5e771bdaf6cc07183ff"} Jul 06 19:34:51.600934 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:51.600611 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gcftt" podStartSLOduration=2.742597919 podStartE2EDuration="25.600594147s" totalImagesPullingTime="22.857996228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:51.599641838 +0000 UTC m=+532.475188530" watchObservedRunningTime="2026-07-06 19:34:51.600594147 +0000 UTC m=+532.476140829" Jul 06 19:34:58.652650 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:58.652618 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:34:58.754166 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:34:58.754128 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:01.391142 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:01.391092 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:01.546348 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:01.546300 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:04.462775 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:04.462744 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:04.564565 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:04.564516 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:08.069348 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:08.069317 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:08.166840 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:08.166808 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:08.702672 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:08.702640 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:09.081387 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:09.081271 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:10.526955 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:10.526924 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:10.640857 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:10.640824 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:10.740497 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:10.740456 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:11.023847 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.023814 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:35:11.030159 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.030141 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:11.032501 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.032482 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:35:11.037899 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.037877 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:35:11.076843 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.076819 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:35:11.215794 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.215761 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45dxp\" (UniqueName: \"kubernetes.io/projected/7c63ff5c-06c6-4341-9e73-92b7e06327ac-kube-api-access-45dxp\") pod \"limitador-limitador-54d6d988d5-6lsqh\" (UID: \"7c63ff5c-06c6-4341-9e73-92b7e06327ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:11.215970 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.215810 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c63ff5c-06c6-4341-9e73-92b7e06327ac-config-file\") pod \"limitador-limitador-54d6d988d5-6lsqh\" (UID: \"7c63ff5c-06c6-4341-9e73-92b7e06327ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:11.316933 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.316849 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45dxp\" (UniqueName: \"kubernetes.io/projected/7c63ff5c-06c6-4341-9e73-92b7e06327ac-kube-api-access-45dxp\") pod \"limitador-limitador-54d6d988d5-6lsqh\" (UID: \"7c63ff5c-06c6-4341-9e73-92b7e06327ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:11.316933 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.316893 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c63ff5c-06c6-4341-9e73-92b7e06327ac-config-file\") pod \"limitador-limitador-54d6d988d5-6lsqh\" (UID: \"7c63ff5c-06c6-4341-9e73-92b7e06327ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:11.317457 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.317412 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c63ff5c-06c6-4341-9e73-92b7e06327ac-config-file\") pod \"limitador-limitador-54d6d988d5-6lsqh\" (UID: \"7c63ff5c-06c6-4341-9e73-92b7e06327ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:11.326962 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.326940 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45dxp\" (UniqueName: \"kubernetes.io/projected/7c63ff5c-06c6-4341-9e73-92b7e06327ac-kube-api-access-45dxp\") pod \"limitador-limitador-54d6d988d5-6lsqh\" (UID: \"7c63ff5c-06c6-4341-9e73-92b7e06327ac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:11.340826 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.340803 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:11.482474 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.482450 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:35:11.485003 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:35:11.484969 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c63ff5c_06c6_4341_9e73_92b7e06327ac.slice/crio-3ec8471944dbf464e32f70c8a27afec12103f39317e69bc72359ded048f940ac WatchSource:0}: Error finding container 3ec8471944dbf464e32f70c8a27afec12103f39317e69bc72359ded048f940ac: Status 404 returned error can't find the container with id 3ec8471944dbf464e32f70c8a27afec12103f39317e69bc72359ded048f940ac Jul 06 19:35:11.555098 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.555066 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:11.635477 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.635445 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" event={"ID":"7c63ff5c-06c6-4341-9e73-92b7e06327ac","Type":"ContainerStarted","Data":"3ec8471944dbf464e32f70c8a27afec12103f39317e69bc72359ded048f940ac"} Jul 06 19:35:11.651590 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.651566 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:11.691705 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.691677 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9bmh"] Jul 06 19:35:11.694812 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.694797 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f9bmh" Jul 06 19:35:11.696981 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.696960 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c5pcx\"" Jul 06 19:35:11.703860 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.703840 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9bmh"] Jul 06 19:35:11.821154 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.821124 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p248k\" (UniqueName: \"kubernetes.io/projected/f5a46acf-b778-40b0-b559-759d4523cb35-kube-api-access-p248k\") pod \"authorino-75d556bd77-f9bmh\" (UID: \"f5a46acf-b778-40b0-b559-759d4523cb35\") " pod="kuadrant-system/authorino-75d556bd77-f9bmh" Jul 06 19:35:11.921907 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.921796 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p248k\" (UniqueName: \"kubernetes.io/projected/f5a46acf-b778-40b0-b559-759d4523cb35-kube-api-access-p248k\") pod \"authorino-75d556bd77-f9bmh\" (UID: \"f5a46acf-b778-40b0-b559-759d4523cb35\") " pod="kuadrant-system/authorino-75d556bd77-f9bmh" Jul 06 19:35:11.931187 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:11.931156 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p248k\" (UniqueName: \"kubernetes.io/projected/f5a46acf-b778-40b0-b559-759d4523cb35-kube-api-access-p248k\") pod \"authorino-75d556bd77-f9bmh\" (UID: \"f5a46acf-b778-40b0-b559-759d4523cb35\") " pod="kuadrant-system/authorino-75d556bd77-f9bmh" Jul 06 19:35:12.004466 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:12.004264 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f9bmh" Jul 06 19:35:12.180944 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:12.180840 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9bmh"] Jul 06 19:35:12.185408 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:35:12.185369 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5a46acf_b778_40b0_b559_759d4523cb35.slice/crio-ae2241ab719454f236e4847435c2634cc3984929ed4059e36738a65a4dcebdcb WatchSource:0}: Error finding container ae2241ab719454f236e4847435c2634cc3984929ed4059e36738a65a4dcebdcb: Status 404 returned error can't find the container with id ae2241ab719454f236e4847435c2634cc3984929ed4059e36738a65a4dcebdcb Jul 06 19:35:12.641173 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:12.641127 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f9bmh" event={"ID":"f5a46acf-b778-40b0-b559-759d4523cb35","Type":"ContainerStarted","Data":"ae2241ab719454f236e4847435c2634cc3984929ed4059e36738a65a4dcebdcb"} Jul 06 19:35:14.652644 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:14.652579 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" event={"ID":"7c63ff5c-06c6-4341-9e73-92b7e06327ac","Type":"ContainerStarted","Data":"095eef3c590cce2cf382477e1a2b770b0c88b5f780d492da3b9eeeb0fe34c5a0"} Jul 06 19:35:14.653208 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:14.652806 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:14.671747 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:14.671644 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" podStartSLOduration=1.931378832 podStartE2EDuration="4.67162385s" totalImagesPullingTime="2.740245018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:14.670683047 +0000 UTC m=+555.546229727" watchObservedRunningTime="2026-07-06 19:35:14.67162385 +0000 UTC m=+555.547170534" Jul 06 19:35:16.662797 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:16.662764 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f9bmh" event={"ID":"f5a46acf-b778-40b0-b559-759d4523cb35","Type":"ContainerStarted","Data":"1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6"} Jul 06 19:35:16.680370 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:16.680326 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-f9bmh" podStartSLOduration=1.7964485890000002 podStartE2EDuration="5.680311699s" totalImagesPullingTime="3.88386311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:35:16.679899092 +0000 UTC m=+557.555445773" watchObservedRunningTime="2026-07-06 19:35:16.680311699 +0000 UTC m=+557.555858379" Jul 06 19:35:17.381162 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:17.381128 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:17.481780 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:17.481745 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:25.658723 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:25.658690 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6lsqh" Jul 06 19:35:32.267846 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:32.267807 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:33.552378 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:33.552349 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:41.048200 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:41.048154 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:43.952866 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:43.952837 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:44.062434 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.062385 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:44.472494 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.472461 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-585c988684-gdnnz"] Jul 06 19:35:44.474494 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.474479 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-585c988684-gdnnz" Jul 06 19:35:44.486045 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.486024 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-585c988684-gdnnz"] Jul 06 19:35:44.564142 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.564118 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fzt\" (UniqueName: \"kubernetes.io/projected/7bae6023-9f3e-461d-a161-d1a4e684aee8-kube-api-access-62fzt\") pod \"authorino-585c988684-gdnnz\" (UID: \"7bae6023-9f3e-461d-a161-d1a4e684aee8\") " pod="kuadrant-system/authorino-585c988684-gdnnz" Jul 06 19:35:44.620724 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.620697 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-585c988684-gdnnz"] Jul 06 19:35:44.620890 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:35:44.620874 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-62fzt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-585c988684-gdnnz" podUID="7bae6023-9f3e-461d-a161-d1a4e684aee8" Jul 06 19:35:44.665332 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.665307 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fzt\" (UniqueName: \"kubernetes.io/projected/7bae6023-9f3e-461d-a161-d1a4e684aee8-kube-api-access-62fzt\") pod \"authorino-585c988684-gdnnz\" (UID: \"7bae6023-9f3e-461d-a161-d1a4e684aee8\") " pod="kuadrant-system/authorino-585c988684-gdnnz" Jul 06 19:35:44.677665 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.677637 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fzt\" (UniqueName: \"kubernetes.io/projected/7bae6023-9f3e-461d-a161-d1a4e684aee8-kube-api-access-62fzt\") pod \"authorino-585c988684-gdnnz\" (UID: \"7bae6023-9f3e-461d-a161-d1a4e684aee8\") " pod="kuadrant-system/authorino-585c988684-gdnnz" Jul 06 19:35:44.760829 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.760765 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-585c988684-gdnnz" Jul 06 19:35:44.766326 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.766309 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-585c988684-gdnnz" Jul 06 19:35:44.866611 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.866579 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62fzt\" (UniqueName: \"kubernetes.io/projected/7bae6023-9f3e-461d-a161-d1a4e684aee8-kube-api-access-62fzt\") pod \"7bae6023-9f3e-461d-a161-d1a4e684aee8\" (UID: \"7bae6023-9f3e-461d-a161-d1a4e684aee8\") " Jul 06 19:35:44.868788 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.868761 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bae6023-9f3e-461d-a161-d1a4e684aee8-kube-api-access-62fzt" (OuterVolumeSpecName: "kube-api-access-62fzt") pod "7bae6023-9f3e-461d-a161-d1a4e684aee8" (UID: "7bae6023-9f3e-461d-a161-d1a4e684aee8"). InnerVolumeSpecName "kube-api-access-62fzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:44.967617 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:44.967588 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62fzt\" (UniqueName: \"kubernetes.io/projected/7bae6023-9f3e-461d-a161-d1a4e684aee8-kube-api-access-62fzt\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:35:45.764629 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:45.764596 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-585c988684-gdnnz" Jul 06 19:35:45.804954 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:45.804923 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-585c988684-gdnnz"] Jul 06 19:35:45.822820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:45.822790 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-585c988684-gdnnz"] Jul 06 19:35:46.681451 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:46.681398 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9bmh"] Jul 06 19:35:46.681901 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:46.681595 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-f9bmh" podUID="f5a46acf-b778-40b0-b559-759d4523cb35" containerName="authorino" containerID="cri-o://1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6" gracePeriod=30 Jul 06 19:35:46.925161 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:46.925134 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f9bmh" Jul 06 19:35:46.980742 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:46.980680 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p248k\" (UniqueName: \"kubernetes.io/projected/f5a46acf-b778-40b0-b559-759d4523cb35-kube-api-access-p248k\") pod \"f5a46acf-b778-40b0-b559-759d4523cb35\" (UID: \"f5a46acf-b778-40b0-b559-759d4523cb35\") " Jul 06 19:35:46.982886 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:46.982861 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a46acf-b778-40b0-b559-759d4523cb35-kube-api-access-p248k" (OuterVolumeSpecName: "kube-api-access-p248k") pod "f5a46acf-b778-40b0-b559-759d4523cb35" (UID: "f5a46acf-b778-40b0-b559-759d4523cb35"). InnerVolumeSpecName "kube-api-access-p248k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:47.081690 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.081655 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p248k\" (UniqueName: \"kubernetes.io/projected/f5a46acf-b778-40b0-b559-759d4523cb35-kube-api-access-p248k\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:35:47.603391 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.603357 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bae6023-9f3e-461d-a161-d1a4e684aee8" path="/var/lib/kubelet/pods/7bae6023-9f3e-461d-a161-d1a4e684aee8/volumes" Jul 06 19:35:47.772467 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.772404 2545 generic.go:356] "Generic (PLEG): container finished" podID="f5a46acf-b778-40b0-b559-759d4523cb35" containerID="1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6" exitCode=0 Jul 06 19:35:47.772874 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.772473 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f9bmh" Jul 06 19:35:47.772874 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.772489 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f9bmh" event={"ID":"f5a46acf-b778-40b0-b559-759d4523cb35","Type":"ContainerDied","Data":"1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6"} Jul 06 19:35:47.772874 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.772530 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f9bmh" event={"ID":"f5a46acf-b778-40b0-b559-759d4523cb35","Type":"ContainerDied","Data":"ae2241ab719454f236e4847435c2634cc3984929ed4059e36738a65a4dcebdcb"} Jul 06 19:35:47.772874 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.772546 2545 scope.go:117] "RemoveContainer" containerID="1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6" Jul 06 19:35:47.785126 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.785089 2545 scope.go:117] "RemoveContainer" containerID="1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6" Jul 06 19:35:47.785566 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:35:47.785548 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6\": container with ID starting with 1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6 not found: ID does not exist" containerID="1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6" Jul 06 19:35:47.785639 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.785573 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6"} err="failed to get container status \"1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6\": rpc error: code = NotFound desc = could not find container \"1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6\": container with ID starting with 1c27d5ff9ec5db6f06d5fd137ef74309d691ad88fbd9ee229fa685e90e9cccd6 not found: ID does not exist" Jul 06 19:35:47.790384 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.790360 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9bmh"] Jul 06 19:35:47.795914 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:47.795896 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f9bmh"] Jul 06 19:35:49.603106 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:49.603077 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a46acf-b778-40b0-b559-759d4523cb35" path="/var/lib/kubelet/pods/f5a46acf-b778-40b0-b559-759d4523cb35/volumes" Jul 06 19:35:51.166653 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:51.166621 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:51.298312 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:51.298273 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:51.375572 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:51.375545 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:54.171542 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:54.171514 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:54.289886 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:54.289858 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:35:59.527383 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:59.527359 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:35:59.533114 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:35:59.533097 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:36:13.081665 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:13.081632 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:14.315726 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:14.315683 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:16.997092 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:16.997062 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:23.103833 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:23.103767 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:23.609479 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:23.609449 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:23.756504 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:23.756478 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:24.211652 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:24.211618 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:24.356923 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:24.356893 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:24.528226 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:24.528145 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:36:34.774543 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:34.774503 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:34.880394 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:34.880360 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:37.926665 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:37.926632 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:48.752891 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:48.752853 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:49.275823 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:49.275788 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:36:54.873960 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:54.873929 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:36:55.643947 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:36:55.643909 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:37:02.864147 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:02.864114 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:04.040748 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:04.040716 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:37:09.730125 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.730095 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd"] Jul 06 19:37:09.730627 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.730484 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a46acf-b778-40b0-b559-759d4523cb35" containerName="authorino" Jul 06 19:37:09.730627 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.730503 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a46acf-b778-40b0-b559-759d4523cb35" containerName="authorino" Jul 06 19:37:09.730627 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.730576 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a46acf-b778-40b0-b559-759d4523cb35" containerName="authorino" Jul 06 19:37:09.733392 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.733367 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.736656 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.736634 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cqnv7\"" Jul 06 19:37:09.736781 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.736634 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:37:09.736781 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.736691 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:37:09.737526 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.737508 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 19:37:09.745931 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.745912 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd"] Jul 06 19:37:09.774406 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.774387 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:09.784533 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.784511 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.784609 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.784566 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxnx\" (UniqueName: \"kubernetes.io/projected/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-kube-api-access-zbxnx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.784667 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.784631 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.784722 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.784677 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.784722 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.784709 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.784815 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.784746 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.885744 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.885715 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.885744 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.885746 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.885928 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.885768 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.885928 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.885810 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxnx\" (UniqueName: \"kubernetes.io/projected/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-kube-api-access-zbxnx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.885928 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.885838 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.885928 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.885859 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.886176 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.886151 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.886300 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.886192 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.886300 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.886289 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.887975 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.887951 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.888357 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.888332 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:09.895933 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:09.895915 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxnx\" (UniqueName: \"kubernetes.io/projected/4b3b95ae-3a83-4638-8cd8-d62b766bc6fe-kube-api-access-zbxnx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd\" (UID: \"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:10.043542 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:10.043478 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:10.178051 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:10.178020 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd"] Jul 06 19:37:10.182909 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:37:10.182886 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3b95ae_3a83_4638_8cd8_d62b766bc6fe.slice/crio-29d8d177f131b5da14bdef889cf1b30750a4ff493e73adea7444e1d0eb0530d5 WatchSource:0}: Error finding container 29d8d177f131b5da14bdef889cf1b30750a4ff493e73adea7444e1d0eb0530d5: Status 404 returned error can't find the container with id 29d8d177f131b5da14bdef889cf1b30750a4ff493e73adea7444e1d0eb0530d5 Jul 06 19:37:11.074297 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:11.074261 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" event={"ID":"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe","Type":"ContainerStarted","Data":"29d8d177f131b5da14bdef889cf1b30750a4ff493e73adea7444e1d0eb0530d5"} Jul 06 19:37:11.244911 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:11.244885 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:37:12.079105 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:12.079067 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" event={"ID":"4b3b95ae-3a83-4638-8cd8-d62b766bc6fe","Type":"ContainerStarted","Data":"173971e3ce93b631021a5498f03bcd6492f4c2d48896c0b55439ff053d69cacb"} Jul 06 19:37:12.104152 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:12.104100 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" podStartSLOduration=2.078008865 podStartE2EDuration="3.104081646s" totalImagesPullingTime="1.026072781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:12.102479768 +0000 UTC m=+672.978026449" watchObservedRunningTime="2026-07-06 19:37:12.104081646 +0000 UTC m=+672.979628331" Jul 06 19:37:20.044032 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:20.043997 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:20.044537 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:20.044054 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:20.056281 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:20.056262 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:20.119009 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:20.118987 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd" Jul 06 19:37:20.979138 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:20.979103 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:21.087305 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:21.087270 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:24.505851 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:24.505821 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:26.083781 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:26.083752 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:26.141522 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:26.141492 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:37:30.137280 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.137249 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg"] Jul 06 19:37:30.140272 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.140243 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.142629 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.142608 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 19:37:30.152844 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.152822 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg"] Jul 06 19:37:30.180502 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.180478 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:30.237474 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.237449 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrln\" (UniqueName: \"kubernetes.io/projected/972c377b-2b56-4831-9716-0b17049c3894-kube-api-access-mkrln\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.237585 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.237492 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972c377b-2b56-4831-9716-0b17049c3894-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.237585 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.237522 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.237585 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.237560 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.237585 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.237581 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.237716 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.237631 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.338726 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.338692 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.338850 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.338739 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.338903 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.338859 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.338953 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.338904 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrln\" (UniqueName: \"kubernetes.io/projected/972c377b-2b56-4831-9716-0b17049c3894-kube-api-access-mkrln\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.339007 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.338958 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972c377b-2b56-4831-9716-0b17049c3894-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.339057 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.339012 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.339129 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.339109 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.339279 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.339255 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.339314 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.339261 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.341141 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.341122 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/972c377b-2b56-4831-9716-0b17049c3894-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.341433 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.341399 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972c377b-2b56-4831-9716-0b17049c3894-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.351348 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.351330 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrln\" (UniqueName: \"kubernetes.io/projected/972c377b-2b56-4831-9716-0b17049c3894-kube-api-access-mkrln\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg\" (UID: \"972c377b-2b56-4831-9716-0b17049c3894\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.449763 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.449689 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:30.578894 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.578804 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg"] Jul 06 19:37:30.581031 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:37:30.580996 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod972c377b_2b56_4831_9716_0b17049c3894.slice/crio-36ec2fa2217e9072578513b0ec90558ae0d66e8119b8c8bfe80c0bd15f6640a8 WatchSource:0}: Error finding container 36ec2fa2217e9072578513b0ec90558ae0d66e8119b8c8bfe80c0bd15f6640a8: Status 404 returned error can't find the container with id 36ec2fa2217e9072578513b0ec90558ae0d66e8119b8c8bfe80c0bd15f6640a8 Jul 06 19:37:30.582331 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:30.582315 2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:37:31.147983 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:31.147953 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" event={"ID":"972c377b-2b56-4831-9716-0b17049c3894","Type":"ContainerStarted","Data":"4bf5f5adc1d829768e11195baf49f8bef740cbd25e7576e3079980112633cc6d"} Jul 06 19:37:31.148357 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:31.147991 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" event={"ID":"972c377b-2b56-4831-9716-0b17049c3894","Type":"ContainerStarted","Data":"36ec2fa2217e9072578513b0ec90558ae0d66e8119b8c8bfe80c0bd15f6640a8"} Jul 06 19:37:33.760672 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:33.760612 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" podStartSLOduration=3.602365573 podStartE2EDuration="3.760590561s" totalImagesPullingTime="158.224988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:31.169931751 +0000 UTC m=+692.045478432" watchObservedRunningTime="2026-07-06 19:37:33.760590561 +0000 UTC m=+694.636137240" Jul 06 19:37:33.762456 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:33.762434 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:37:39.131176 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:39.131141 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:37:40.450671 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:40.450641 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:40.450671 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:40.450678 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:40.463138 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:40.463114 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:37:41.197480 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:37:41.197453 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg" Jul 06 19:38:28.028601 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.028564 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bc667b49-sj47w"] Jul 06 19:38:28.032494 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.032467 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bc667b49-sj47w" Jul 06 19:38:28.035286 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.035268 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 19:38:28.036144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.036126 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c5pcx\"" Jul 06 19:38:28.040072 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.040051 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bc667b49-sj47w"] Jul 06 19:38:28.052353 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.052328 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f53f4bfd-437d-4650-a445-2f09705909ba-tls-cert\") pod \"authorino-bc667b49-sj47w\" (UID: \"f53f4bfd-437d-4650-a445-2f09705909ba\") " pod="kuadrant-system/authorino-bc667b49-sj47w" Jul 06 19:38:28.052483 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.052465 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5hhp\" (UniqueName: \"kubernetes.io/projected/f53f4bfd-437d-4650-a445-2f09705909ba-kube-api-access-p5hhp\") pod \"authorino-bc667b49-sj47w\" (UID: \"f53f4bfd-437d-4650-a445-2f09705909ba\") " pod="kuadrant-system/authorino-bc667b49-sj47w" Jul 06 19:38:28.153235 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.153198 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hhp\" (UniqueName: \"kubernetes.io/projected/f53f4bfd-437d-4650-a445-2f09705909ba-kube-api-access-p5hhp\") pod \"authorino-bc667b49-sj47w\" (UID: \"f53f4bfd-437d-4650-a445-2f09705909ba\") " pod="kuadrant-system/authorino-bc667b49-sj47w" Jul 06 19:38:28.153235 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.153236 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f53f4bfd-437d-4650-a445-2f09705909ba-tls-cert\") pod \"authorino-bc667b49-sj47w\" (UID: \"f53f4bfd-437d-4650-a445-2f09705909ba\") " pod="kuadrant-system/authorino-bc667b49-sj47w" Jul 06 19:38:28.155768 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.155744 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f53f4bfd-437d-4650-a445-2f09705909ba-tls-cert\") pod \"authorino-bc667b49-sj47w\" (UID: \"f53f4bfd-437d-4650-a445-2f09705909ba\") " pod="kuadrant-system/authorino-bc667b49-sj47w" Jul 06 19:38:28.163405 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.163387 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hhp\" (UniqueName: \"kubernetes.io/projected/f53f4bfd-437d-4650-a445-2f09705909ba-kube-api-access-p5hhp\") pod \"authorino-bc667b49-sj47w\" (UID: \"f53f4bfd-437d-4650-a445-2f09705909ba\") " pod="kuadrant-system/authorino-bc667b49-sj47w" Jul 06 19:38:28.342317 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.342247 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bc667b49-sj47w" Jul 06 19:38:28.433100 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.433072 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:28.471691 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.471662 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bc667b49-sj47w"] Jul 06 19:38:28.475485 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:38:28.475450 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf53f4bfd_437d_4650_a445_2f09705909ba.slice/crio-fe461a09dea1419498f64dd0897458daa10dc1f81709773ce140e06d47d4398d WatchSource:0}: Error finding container fe461a09dea1419498f64dd0897458daa10dc1f81709773ce140e06d47d4398d: Status 404 returned error can't find the container with id fe461a09dea1419498f64dd0897458daa10dc1f81709773ce140e06d47d4398d Jul 06 19:38:28.536188 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:28.536164 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:29.369453 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:29.369343 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bc667b49-sj47w" event={"ID":"f53f4bfd-437d-4650-a445-2f09705909ba","Type":"ContainerStarted","Data":"11aa0ccd84901895d60375f0a5b4aa270fdd45ecd57801bd68657b0776f516f2"} Jul 06 19:38:29.369453 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:29.369394 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bc667b49-sj47w" event={"ID":"f53f4bfd-437d-4650-a445-2f09705909ba","Type":"ContainerStarted","Data":"fe461a09dea1419498f64dd0897458daa10dc1f81709773ce140e06d47d4398d"} Jul 06 19:38:29.634506 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:29.634373 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:29.731164 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:29.731128 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:38:57.685070 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:57.685015 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bc667b49-sj47w" podStartSLOduration=29.270128616 podStartE2EDuration="29.684998492s" totalImagesPullingTime="414.869876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:29.391635451 +0000 UTC m=+750.267182135" watchObservedRunningTime="2026-07-06 19:38:57.684998492 +0000 UTC m=+778.560545210" Jul 06 19:38:57.685776 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:38:57.685754 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:39:07.134205 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:07.134168 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:39:19.049024 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:19.048990 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:39:26.620058 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:26.619972 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:27.916616 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:27.916581 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:28.132447 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:28.132398 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:39:37.269627 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:37.269589 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:37.367987 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:37.367929 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:37.451957 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:37.451928 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:38.040954 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:38.040926 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:39:41.030878 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:41.030847 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:39:42.873955 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:39:42.873913 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:19.098406 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:19.098349 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:19.245315 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:19.245287 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:19.372478 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:19.372381 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:25.844109 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:25.844079 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:25.995146 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:25.995118 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:26.129506 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:26.129433 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:26.276071 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:26.276040 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:37.519431 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:37.519383 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:38.070002 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:38.069973 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:40:49.142228 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:49.142193 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:40:55.935944 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:55.935865 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:40:59.550002 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:59.549976 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:40:59.553130 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:40:59.553109 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:41:06.140174 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:06.140147 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:41:31.067143 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:31.067109 2545 generic.go:356] "Generic (PLEG): container finished" podID="6c092ad3-3008-4a3e-bf03-963b4d69388a" containerID="65f10507d4191ade8b4b5b8b7b66cda8f9dd6e244cf6c724c7c858a4b2841444" exitCode=1 Jul 06 19:41:31.067575 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:31.067181 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" event={"ID":"6c092ad3-3008-4a3e-bf03-963b4d69388a","Type":"ContainerDied","Data":"65f10507d4191ade8b4b5b8b7b66cda8f9dd6e244cf6c724c7c858a4b2841444"} Jul 06 19:41:31.067575 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:31.067505 2545 scope.go:117] "RemoveContainer" containerID="65f10507d4191ade8b4b5b8b7b66cda8f9dd6e244cf6c724c7c858a4b2841444" Jul 06 19:41:32.072518 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:32.072482 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" event={"ID":"6c092ad3-3008-4a3e-bf03-963b4d69388a","Type":"ContainerStarted","Data":"fa756e18c48c6bc924e652776923ffee133321ac961a37a22385b4fd2638ded7"} Jul 06 19:41:32.072897 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:32.072687 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:41:39.152185 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:39.152153 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:42.283797 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:42.283754 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:43.079312 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:43.079280 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-94489db4c-b5dqb" Jul 06 19:41:43.571004 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:43.570972 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:41:55.836097 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:41:55.836066 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:42:05.254816 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:42:05.254782 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:42:15.056196 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:42:15.056163 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:42:37.338523 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:42:37.338444 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:42:46.958064 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:42:46.958027 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:43:10.550729 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:43:10.550694 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:43:16.327461 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:43:16.327411 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:43:20.840785 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:43:20.840745 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:43:36.537841 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:43:36.537805 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:43:45.586944 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:43:45.586901 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:43:56.240769 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:43:56.240691 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:44:05.179892 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:44:05.179859 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:44:13.434725 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:44:13.434689 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:45:22.741871 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:45:22.741798 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:33.334604 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:45:33.334568 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:49.606880 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:45:49.606823 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:49.700860 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:45:49.700830 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:45:59.576925 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:45:59.576891 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:45:59.577359 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:45:59.576935 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:46:22.277670 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:46:22.277632 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:46:30.028886 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:46:30.028855 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:46:43.131343 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:46:43.131307 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:46:46.895796 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:46:46.895760 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:46.999207 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:46:46.999160 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:46:51.342262 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:46:51.342180 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:47:00.427577 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:47:00.427544 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:47:08.429822 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:47:08.429794 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:47:18.924074 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:47:18.924037 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:47:25.924876 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:47:25.924844 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:47:34.941828 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:47:34.941791 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:47:44.424485 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:47:44.424451 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:47:51.728036 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:47:51.728005 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:48:00.231951 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:48:00.231915 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:48:09.334527 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:48:09.334494 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:48:28.129001 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:48:28.128923 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:48:36.825463 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:48:36.825409 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:48:46.632810 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:48:46.632777 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:48:57.238450 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:48:57.238398 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:49:20.835367 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:20.835330 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:49:29.529032 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:29.529002 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:49:39.229580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:39.229536 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:49:47.230972 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:47.230935 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:47.308732 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:47.308702 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:48.635840 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:48.635803 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:49:52.168807 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:52.168728 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:52.270030 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:52.270002 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:57.169692 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:57.169655 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:57.236941 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:57.236910 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:49:58.726450 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:49:58.726406 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:50:02.245476 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:02.245411 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:02.358639 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:02.358609 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:07.214613 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:07.214574 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:07.315494 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:07.315464 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:09.732549 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:09.732515 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:09.832615 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:09.832579 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:09.934588 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:09.934563 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:10.034635 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:10.034563 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:10.135767 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:10.135738 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:10.206623 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:10.206598 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:11.475413 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:11.475380 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:11.576333 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:11.576302 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:11.732856 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:11.732771 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:50:12.068570 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:12.068491 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:12.168815 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:12.168774 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:20.841334 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:20.841295 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:50:31.837126 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:31.837084 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:50:42.830981 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:42.830949 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:50:50.833974 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:50.833941 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:52.208687 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:52.208659 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:50:53.640401 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:53.640367 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:50:59.599725 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:59.599689 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:50:59.600496 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:50:59.600474 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:51:02.123447 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:51:02.123400 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:51:21.329202 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:51:21.329127 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:51:33.124238 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:51:33.124195 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:51:44.731052 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:51:44.731016 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:51:46.769495 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:51:46.769458 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:52.224968 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:51:52.224935 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:51:56.823138 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:51:56.823105 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:52:00.528912 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:00.528881 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:04.728099 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:04.728069 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:52:05.713808 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:05.713776 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:08.637558 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:08.637526 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:08.736146 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:08.736112 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:09.129826 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:09.129737 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:13.831021 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:13.830992 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:52:15.665796 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:15.665767 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:15.790296 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:15.790266 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:15.969791 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:15.969686 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:16.132254 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:16.132224 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:17.146154 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:17.146123 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:17.243766 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:17.243737 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:17.349716 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:17.349683 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:17.697136 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:17.697099 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:17.878558 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:17.878527 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:18.056622 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:18.056537 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:19.755212 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:19.755183 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:22.191002 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:22.190972 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:22.374648 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:22.374618 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:22.476302 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:22.476207 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:22.607091 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:22.607061 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:24.371639 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:24.371608 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:29.038822 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:29.038792 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:29.113011 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:29.112986 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:29.320295 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:29.320178 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:29.419820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:29.419792 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:31.290187 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:31.290152 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:31.373228 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:31.373204 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:32.828669 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:32.828643 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:52:32.838139 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:32.838120 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:35.599257 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:35.599224 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:35.956233 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:35.956202 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:41.724754 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:41.724720 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:41.849504 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:41.849474 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:42.046169 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:42.046076 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:52:48.891343 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:48.891310 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:48.967455 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:48.967406 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:49.122824 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:49.122793 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:49.223037 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:49.223006 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:49.324950 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:49.324916 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:50.274730 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:50.274700 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:50.455811 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:50.455731 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:50.583622 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:50.583541 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:50.719298 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:50.719269 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:51.042048 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:51.042016 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:51.134654 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:51.134620 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:53.134841 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:53.134810 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:53.227575 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:53.227545 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:52:54.683791 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:54.683755 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:54.876200 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:54.876174 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:54.968469 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:54.968372 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:55.091014 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:55.090979 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:52:56.462327 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:52:56.462294 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:01.749019 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:01.748975 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:02.524003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:02.523970 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:53:12.983163 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:12.983131 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:17.201394 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:17.201358 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:17.324899 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:17.324871 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:20.064430 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:20.064385 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:20.165610 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:20.165579 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:22.192023 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:22.191991 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:25.045028 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:25.044995 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:25.147603 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:25.147569 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:30.059608 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:30.059575 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:30.122886 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:30.122857 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:32.205100 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:32.205073 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:35.064003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:35.063970 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:35.146243 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:35.146213 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:37.211225 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:37.211194 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:40.079728 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:40.079695 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:40.164680 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:40.164646 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:41.668944 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:41.668915 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:41.770573 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:41.770539 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:41.994744 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:41.994669 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:42.063815 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:42.063780 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:42.127685 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:42.127653 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:53:43.147059 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:53:43.147032 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:20.899757 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:20.899666 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:20.996530 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:20.996498 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:23.514316 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:23.514286 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:23.637810 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:23.637776 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:24.264883 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:24.264847 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:25.210623 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:25.210583 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:25.341563 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:25.341531 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:25.517667 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:25.517583 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:25.654848 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:25.654821 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:27.262619 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:27.262587 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:35.691056 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:35.691021 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:35.762690 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:35.762643 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:35.848783 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:35.848749 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:35.947596 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:35.947505 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:36.050259 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:36.050224 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:54:36.459149 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:54:36.459112 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:02.486881 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:02.486844 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:09.151598 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:09.151558 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:09.274300 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:09.274271 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:09.372150 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:09.372120 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:25.249483 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:25.249445 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:31.385856 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:31.385824 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:31.491849 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:31.491814 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:31.586013 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:31.585979 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:31.688140 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:31.688047 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:42.139520 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:42.139486 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:42.942360 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:42.942329 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:49.230937 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:49.230891 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:49.347650 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:49.347614 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:49.478197 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:49.478165 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:49.615796 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:49.615715 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:50.372234 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:50.372202 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:50.502541 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:50.502453 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:50.658943 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:50.658909 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:51.530746 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:51.530714 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:55:52.255205 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:52.255172 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:58.158406 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:58.158368 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:58.270693 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:58.270667 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:58.554086 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:58.554005 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:55:58.864255 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:58.864180 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:55:59.625757 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:59.625726 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:55:59.628161 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:55:59.628139 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 19:56:05.070902 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:05.070864 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:05.147349 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:05.147321 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:05.317436 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:05.317389 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:05.657007 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:05.656974 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:05.806924 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:05.806892 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:06.222144 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:06.222110 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:56:08.416821 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:08.416794 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:08.567966 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:08.567921 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:08.687090 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:08.687018 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:08.883432 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:08.883389 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:09.333636 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:09.333601 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:09.466074 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:09.466045 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:12.956518 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:12.956483 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:13.620749 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:13.620720 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:13.724738 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:13.724710 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:13.832675 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:13.832643 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:56:14.089251 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:14.089190 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:15.658168 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:15.658131 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:20.549823 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:20.549790 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:20.670239 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:20.670207 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:21.004142 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:21.004114 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:21.410145 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:21.410115 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:21.548130 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:21.548101 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:26.625653 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:26.625620 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:56:27.626406 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:27.626376 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:27.706778 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:27.706749 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:28.009705 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:28.009678 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:28.201704 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:28.201671 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:28.322139 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:28.322069 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:28.422140 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:28.422117 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:28.623619 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:28.623538 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:34.385232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:34.385201 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:34.454809 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:34.454778 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:34.534478 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:34.534450 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:34.635150 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:34.635120 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:34.761860 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:34.761784 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:40.906595 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:40.906566 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:41.010885 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:41.010859 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:47.818255 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:47.818207 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:48.172203 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:48.172159 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:48.472029 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:48.471945 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:51.452278 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:51.452248 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:51.547740 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:51.547708 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:51.644795 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:51.644767 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:51.771128 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:51.771049 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:52.022976 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:52.022900 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:52.123992 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:52.123963 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:55.954232 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:55.954199 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:56:57.428947 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:56:57.428910 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:57:05.066367 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:05.066331 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:05.166338 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:05.166306 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:06.321208 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.321171 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz"] Jul 06 19:57:06.324235 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.324217 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.327183 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.327160 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9c52a789\"/\"openshift-service-ca.crt\"" Jul 06 19:57:06.327183 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.327166 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9c52a789\"/\"default-dockercfg-dlpbg\"" Jul 06 19:57:06.327359 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.327189 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9c52a789\"/\"kube-root-ca.crt\"" Jul 06 19:57:06.327359 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.327226 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9c52a789\"/\"e2e-lifecycle-model-9c52a789-kserve-self-signed-certs\"" Jul 06 19:57:06.334615 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.334594 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz"] Jul 06 19:57:06.349313 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.349286 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:06.442700 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.442667 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-model-cache\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.442700 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.442704 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-home\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.442922 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.442738 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-tmp-dir\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.442922 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.442825 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63536c8e-669b-4459-89b6-217d7067f950-tls-certs\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.442922 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.442859 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxkm\" (UniqueName: \"kubernetes.io/projected/63536c8e-669b-4459-89b6-217d7067f950-kube-api-access-dgxkm\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.442922 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.442892 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-dshm\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.543853 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.543817 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63536c8e-669b-4459-89b6-217d7067f950-tls-certs\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.543853 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.543851 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxkm\" (UniqueName: \"kubernetes.io/projected/63536c8e-669b-4459-89b6-217d7067f950-kube-api-access-dgxkm\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.544080 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.543872 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-dshm\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.544080 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.543895 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-model-cache\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.544080 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.543916 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-home\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.544080 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.543971 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-tmp-dir\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.544395 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.544362 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-tmp-dir\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.544395 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.544389 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-model-cache\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.544555 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.544446 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-home\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.546228 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.546209 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-dshm\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.546395 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.546376 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63536c8e-669b-4459-89b6-217d7067f950-tls-certs\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.552659 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.552637 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxkm\" (UniqueName: \"kubernetes.io/projected/63536c8e-669b-4459-89b6-217d7067f950-kube-api-access-dgxkm\") pod \"e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.632943 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.632852 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:06.761115 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:06.760960 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz"] Jul 06 19:57:06.763569 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:57:06.763539 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63536c8e_669b_4459_89b6_217d7067f950.slice/crio-a2b9d94ba57bb654c0a2708c37b5e29a661dad2b80b773c934aedb67e9962237 WatchSource:0}: Error finding container a2b9d94ba57bb654c0a2708c37b5e29a661dad2b80b773c934aedb67e9962237: Status 404 returned error can't find the container with id a2b9d94ba57bb654c0a2708c37b5e29a661dad2b80b773c934aedb67e9962237 Jul 06 19:57:07.762374 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:07.762334 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" event={"ID":"63536c8e-669b-4459-89b6-217d7067f950","Type":"ContainerStarted","Data":"cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b"} Jul 06 19:57:07.762757 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:07.762385 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" event={"ID":"63536c8e-669b-4459-89b6-217d7067f950","Type":"ContainerStarted","Data":"a2b9d94ba57bb654c0a2708c37b5e29a661dad2b80b773c934aedb67e9962237"} Jul 06 19:57:07.779581 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:07.779534 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" podStartSLOduration=1.779519656 podStartE2EDuration="1.779519656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:57:07.77892161 +0000 UTC m=+1868.654468292" watchObservedRunningTime="2026-07-06 19:57:07.779519656 +0000 UTC m=+1868.655066337" Jul 06 19:57:09.136393 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:09.136360 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:57:13.635280 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:13.635227 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:13.734071 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:13.734019 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:16.633321 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:16.633286 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:16.633778 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:16.633335 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:16.645787 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:16.645759 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:16.808965 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:16.808931 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:20.520319 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:20.520240 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:57:23.331515 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:23.331483 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:23.675595 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:23.675552 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:29.413842 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:29.413805 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:32.325137 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:32.325103 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:57:34.390152 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:34.390105 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz"] Jul 06 19:57:34.483275 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:34.483232 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:57:49.412178 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.412127 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" podUID="63536c8e-669b-4459-89b6-217d7067f950" containerName="main" containerID="cri-o://cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b" gracePeriod=45 Jul 06 19:57:49.656116 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.656095 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:49.767396 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767302 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-model-cache\") pod \"63536c8e-669b-4459-89b6-217d7067f950\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " Jul 06 19:57:49.767396 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767366 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgxkm\" (UniqueName: \"kubernetes.io/projected/63536c8e-669b-4459-89b6-217d7067f950-kube-api-access-dgxkm\") pod \"63536c8e-669b-4459-89b6-217d7067f950\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " Jul 06 19:57:49.767396 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767396 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-dshm\") pod \"63536c8e-669b-4459-89b6-217d7067f950\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " Jul 06 19:57:49.767692 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767495 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63536c8e-669b-4459-89b6-217d7067f950-tls-certs\") pod \"63536c8e-669b-4459-89b6-217d7067f950\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " Jul 06 19:57:49.767692 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767544 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-tmp-dir\") pod \"63536c8e-669b-4459-89b6-217d7067f950\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " Jul 06 19:57:49.767692 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767579 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-home\") pod \"63536c8e-669b-4459-89b6-217d7067f950\" (UID: \"63536c8e-669b-4459-89b6-217d7067f950\") " Jul 06 19:57:49.767692 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767649 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-model-cache" (OuterVolumeSpecName: "model-cache") pod "63536c8e-669b-4459-89b6-217d7067f950" (UID: "63536c8e-669b-4459-89b6-217d7067f950"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:49.767897 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767866 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:57:49.767897 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767861 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63536c8e-669b-4459-89b6-217d7067f950" (UID: "63536c8e-669b-4459-89b6-217d7067f950"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:49.768000 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.767897 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-home" (OuterVolumeSpecName: "home") pod "63536c8e-669b-4459-89b6-217d7067f950" (UID: "63536c8e-669b-4459-89b6-217d7067f950"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:49.769694 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.769674 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-dshm" (OuterVolumeSpecName: "dshm") pod "63536c8e-669b-4459-89b6-217d7067f950" (UID: "63536c8e-669b-4459-89b6-217d7067f950"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:49.769694 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.769671 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63536c8e-669b-4459-89b6-217d7067f950-kube-api-access-dgxkm" (OuterVolumeSpecName: "kube-api-access-dgxkm") pod "63536c8e-669b-4459-89b6-217d7067f950" (UID: "63536c8e-669b-4459-89b6-217d7067f950"). InnerVolumeSpecName "kube-api-access-dgxkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:57:49.769839 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.769695 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63536c8e-669b-4459-89b6-217d7067f950-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63536c8e-669b-4459-89b6-217d7067f950" (UID: "63536c8e-669b-4459-89b6-217d7067f950"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:57:49.868820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.868797 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:57:49.868820 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.868817 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63536c8e-669b-4459-89b6-217d7067f950-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:57:49.868959 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.868827 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:57:49.868959 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.868834 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63536c8e-669b-4459-89b6-217d7067f950-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:57:49.868959 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.868842 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgxkm\" (UniqueName: \"kubernetes.io/projected/63536c8e-669b-4459-89b6-217d7067f950-kube-api-access-dgxkm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 19:57:49.928520 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.928493 2545 generic.go:356] "Generic (PLEG): container finished" podID="63536c8e-669b-4459-89b6-217d7067f950" containerID="cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b" exitCode=0 Jul 06 19:57:49.928639 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.928537 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" Jul 06 19:57:49.928639 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.928569 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" event={"ID":"63536c8e-669b-4459-89b6-217d7067f950","Type":"ContainerDied","Data":"cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b"} Jul 06 19:57:49.928639 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.928599 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz" event={"ID":"63536c8e-669b-4459-89b6-217d7067f950","Type":"ContainerDied","Data":"a2b9d94ba57bb654c0a2708c37b5e29a661dad2b80b773c934aedb67e9962237"} Jul 06 19:57:49.928639 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.928614 2545 scope.go:117] "RemoveContainer" containerID="cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b" Jul 06 19:57:49.936998 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.936980 2545 scope.go:117] "RemoveContainer" containerID="cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b" Jul 06 19:57:49.937229 ip-10-0-136-156 kubenswrapper[2545]: E0706 19:57:49.937209 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b\": container with ID starting with cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b not found: ID does not exist" containerID="cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b" Jul 06 19:57:49.937279 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.937236 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b"} err="failed to get container status \"cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b\": rpc error: code = NotFound desc = could not find container \"cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b\": container with ID starting with cf084fd1ba07ef4cb82ec1d1c6ac61000228db0371813a52d9b3e4c9e60e392b not found: ID does not exist" Jul 06 19:57:49.954758 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.954730 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz"] Jul 06 19:57:49.963037 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:49.963017 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9c52a789/e2e-lifecycle-model-9c52a789-kserve-597b576987-mmvpz"] Jul 06 19:57:51.604530 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:57:51.604495 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63536c8e-669b-4459-89b6-217d7067f950" path="/var/lib/kubelet/pods/63536c8e-669b-4459-89b6-217d7067f950/volumes" Jul 06 19:58:11.764176 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:11.764138 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:12.284193 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:12.284159 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:14.234633 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:14.234600 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:58:19.301845 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:19.301812 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:19.410141 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:19.410091 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:20.108273 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:20.108217 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:20.320352 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:20.320324 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:21.369850 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:21.369799 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:21.470108 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:21.470079 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:21.569966 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:21.569941 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:22.319114 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:22.319078 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:22.421266 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:22.421237 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:22.522599 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:22.522575 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:23.990010 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:23.989968 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:24.926446 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:24.926372 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:58:30.409579 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:30.409549 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:30.474967 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:30.474915 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:31.129662 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:31.129631 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:34.617486 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:34.617448 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:58:37.325347 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:37.325311 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:37.457160 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:37.457128 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:37.550023 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:37.549997 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:48.043909 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:48.043876 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:48.384205 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:48.384124 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:58:50.969030 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:58:50.968955 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:09.410738 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:09.410708 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:15.644682 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:15.644647 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:15.761619 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:15.761589 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:16.003199 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:16.003177 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:16.331187 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:16.331112 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:17.067986 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:17.067926 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:19.185086 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:19.185058 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:19.308926 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:19.308899 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:19.431463 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:19.431410 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:20.010202 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:20.010174 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:20.153948 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:20.153917 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:23.789227 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:23.789191 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.078483 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:24.078395 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.413874 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:24.413841 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:24.636868 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:24.636840 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:26.820937 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:26.820903 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:59:27.518462 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:27.518412 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:27.609470 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:27.609436 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:27.741386 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:27.741354 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:27.842979 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:27.842906 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:28.298266 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:28.298240 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:28.400200 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:28.400171 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:32.045001 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:32.044970 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:37.530538 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:37.530500 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:59:40.726872 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.726836 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8"] Jul 06 19:59:40.727314 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.727114 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63536c8e-669b-4459-89b6-217d7067f950" containerName="main" Jul 06 19:59:40.727314 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.727124 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="63536c8e-669b-4459-89b6-217d7067f950" containerName="main" Jul 06 19:59:40.727314 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.727187 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="63536c8e-669b-4459-89b6-217d7067f950" containerName="main" Jul 06 19:59:40.730098 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.730080 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.732259 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.732234 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d188f8\"/\"openshift-service-ca.crt\"" Jul 06 19:59:40.732388 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.732275 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d188f8\"/\"kube-root-ca.crt\"" Jul 06 19:59:40.732869 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.732851 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d188f8\"/\"default-dockercfg-gts6r\"" Jul 06 19:59:40.732981 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.732968 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d188f8\"/\"test-model-d188f8-kserve-self-signed-certs\"" Jul 06 19:59:40.746657 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.746637 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8"] Jul 06 19:59:40.844784 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.844752 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:40.876950 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.876916 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48dn9\" (UniqueName: \"kubernetes.io/projected/89c834f6-2dc3-496b-8add-58ea612b0e5c-kube-api-access-48dn9\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.877091 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.876979 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-home\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.877091 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.877016 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c834f6-2dc3-496b-8add-58ea612b0e5c-tls-certs\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.877091 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.877033 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-model-cache\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.877091 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.877053 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-dshm\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.877222 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.877106 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-tmp-dir\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.947214 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.947181 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:40.978412 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978350 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48dn9\" (UniqueName: \"kubernetes.io/projected/89c834f6-2dc3-496b-8add-58ea612b0e5c-kube-api-access-48dn9\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.978412 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978394 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-home\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.978580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978461 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c834f6-2dc3-496b-8add-58ea612b0e5c-tls-certs\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.978580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978480 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-model-cache\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.978580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978500 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-dshm\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.978580 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978515 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-tmp-dir\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.978887 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978851 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-home\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.979003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978940 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-tmp-dir\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.979003 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.978974 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-model-cache\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.980940 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.980915 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-dshm\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.981103 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.981087 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c834f6-2dc3-496b-8add-58ea612b0e5c-tls-certs\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:40.987189 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:40.987166 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48dn9\" (UniqueName: \"kubernetes.io/projected/89c834f6-2dc3-496b-8add-58ea612b0e5c-kube-api-access-48dn9\") pod \"test-model-d188f8-kserve-778859bf9f-cjgj8\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:41.039195 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:41.039170 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:41.163371 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:41.163333 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8"] Jul 06 19:59:41.166188 ip-10-0-136-156 kubenswrapper[2545]: W0706 19:59:41.166157 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89c834f6_2dc3_496b_8add_58ea612b0e5c.slice/crio-a0c3a2bb8e8bbf485eda4712008be2b1d6ba64d78fdd7bbd1a54003257b6dbb6 WatchSource:0}: Error finding container a0c3a2bb8e8bbf485eda4712008be2b1d6ba64d78fdd7bbd1a54003257b6dbb6: Status 404 returned error can't find the container with id a0c3a2bb8e8bbf485eda4712008be2b1d6ba64d78fdd7bbd1a54003257b6dbb6 Jul 06 19:59:41.367164 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:41.367071 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" event={"ID":"89c834f6-2dc3-496b-8add-58ea612b0e5c","Type":"ContainerStarted","Data":"a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef"} Jul 06 19:59:41.367164 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:41.367123 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" event={"ID":"89c834f6-2dc3-496b-8add-58ea612b0e5c","Type":"ContainerStarted","Data":"a0c3a2bb8e8bbf485eda4712008be2b1d6ba64d78fdd7bbd1a54003257b6dbb6"} Jul 06 19:59:41.384069 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:41.384021 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" podStartSLOduration=1.384004814 podStartE2EDuration="1.384004814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:41.38294311 +0000 UTC m=+2022.258489792" watchObservedRunningTime="2026-07-06 19:59:41.384004814 +0000 UTC m=+2022.259551473" Jul 06 19:59:46.636070 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:46.636023 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 19:59:49.159233 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:49.159203 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:49.259222 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:49.259184 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 19:59:51.039577 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:51.039533 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:51.040011 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:51.039611 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:51.052537 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:51.052516 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:51.419272 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:51.419242 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 19:59:59.834551 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:59.834521 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh"] Jul 06 19:59:59.837786 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:59.837769 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 19:59:59.840813 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:59.840777 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-62de5f\"/\"test-model-62de5f-kserve-self-signed-certs\"" Jul 06 19:59:59.840813 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:59.840807 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-62de5f\"/\"default-dockercfg-25wmv\"" Jul 06 19:59:59.840989 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:59.840834 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-62de5f\"/\"kube-root-ca.crt\"" Jul 06 19:59:59.840989 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:59.840806 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-62de5f\"/\"openshift-service-ca.crt\"" Jul 06 19:59:59.851973 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:59.851953 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh"] Jul 06 19:59:59.885527 ip-10-0-136-156 kubenswrapper[2545]: I0706 19:59:59.885503 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:00.024513 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.024483 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvz5s\" (UniqueName: \"kubernetes.io/projected/0f180001-34c8-4979-8d2a-479579571d18-kube-api-access-wvz5s\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.024666 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.024522 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f180001-34c8-4979-8d2a-479579571d18-tls-certs\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.024666 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.024550 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-model-cache\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.024666 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.024604 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-tmp-dir\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.024817 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.024666 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-dshm\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.024817 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.024705 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-home\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.125773 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.125702 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-dshm\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.125773 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.125732 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-home\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.125971 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.125770 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvz5s\" (UniqueName: \"kubernetes.io/projected/0f180001-34c8-4979-8d2a-479579571d18-kube-api-access-wvz5s\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.125971 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.125801 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f180001-34c8-4979-8d2a-479579571d18-tls-certs\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.125971 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.125831 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-model-cache\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.125971 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.125869 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-tmp-dir\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.126166 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.126113 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-home\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.126209 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.126192 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-tmp-dir\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.126253 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.126236 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-model-cache\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.128193 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.128171 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-dshm\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.128380 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.128367 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f180001-34c8-4979-8d2a-479579571d18-tls-certs\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.139856 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.139835 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvz5s\" (UniqueName: \"kubernetes.io/projected/0f180001-34c8-4979-8d2a-479579571d18-kube-api-access-wvz5s\") pod \"test-model-62de5f-kserve-6c6777cbb9-k4wxh\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.146721 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.146701 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:00.275996 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.275965 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh"] Jul 06 20:00:00.279594 ip-10-0-136-156 kubenswrapper[2545]: W0706 20:00:00.279567 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f180001_34c8_4979_8d2a_479579571d18.slice/crio-74cf20be86a52c0474e1d33636f7e721579ad4e0a89ffd92d2be2fce390e523e WatchSource:0}: Error finding container 74cf20be86a52c0474e1d33636f7e721579ad4e0a89ffd92d2be2fce390e523e: Status 404 returned error can't find the container with id 74cf20be86a52c0474e1d33636f7e721579ad4e0a89ffd92d2be2fce390e523e Jul 06 20:00:00.453525 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.453491 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" event={"ID":"0f180001-34c8-4979-8d2a-479579571d18","Type":"ContainerStarted","Data":"837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2"} Jul 06 20:00:00.453728 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.453537 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" event={"ID":"0f180001-34c8-4979-8d2a-479579571d18","Type":"ContainerStarted","Data":"74cf20be86a52c0474e1d33636f7e721579ad4e0a89ffd92d2be2fce390e523e"} Jul 06 20:00:00.470210 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:00.470168 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" podStartSLOduration=1.470154947 podStartE2EDuration="1.470154947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:00.469074558 +0000 UTC m=+2041.344621239" watchObservedRunningTime="2026-07-06 20:00:00.470154947 +0000 UTC m=+2041.345701663" Jul 06 20:00:03.932595 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:03.932558 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:00:10.147106 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:10.147071 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:10.147516 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:10.147126 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:10.159399 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:10.159378 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:10.504915 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:10.504829 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:16.128221 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:16.128184 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:00:20.757312 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:20.757231 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8"] Jul 06 20:00:20.785891 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:20.785861 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:21.706865 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:21.706834 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh"] Jul 06 20:00:21.730939 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:21.730912 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:21.944915 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:21.944871 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:22.371679 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:22.371644 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:22.468953 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:22.468924 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:22.658589 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:22.658554 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:27.326030 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:27.325997 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:00:28.255454 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:28.255409 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:00:35.782025 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:35.781973 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" podUID="89c834f6-2dc3-496b-8add-58ea612b0e5c" containerName="main" containerID="cri-o://a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef" gracePeriod=45 Jul 06 20:00:36.021937 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.021915 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 20:00:36.116454 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116369 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48dn9\" (UniqueName: \"kubernetes.io/projected/89c834f6-2dc3-496b-8add-58ea612b0e5c-kube-api-access-48dn9\") pod \"89c834f6-2dc3-496b-8add-58ea612b0e5c\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " Jul 06 20:00:36.116454 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116435 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-model-cache\") pod \"89c834f6-2dc3-496b-8add-58ea612b0e5c\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " Jul 06 20:00:36.116643 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116456 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-dshm\") pod \"89c834f6-2dc3-496b-8add-58ea612b0e5c\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " Jul 06 20:00:36.116643 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116488 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c834f6-2dc3-496b-8add-58ea612b0e5c-tls-certs\") pod \"89c834f6-2dc3-496b-8add-58ea612b0e5c\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " Jul 06 20:00:36.116643 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116513 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-home\") pod \"89c834f6-2dc3-496b-8add-58ea612b0e5c\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " Jul 06 20:00:36.116643 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116534 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-tmp-dir\") pod \"89c834f6-2dc3-496b-8add-58ea612b0e5c\" (UID: \"89c834f6-2dc3-496b-8add-58ea612b0e5c\") " Jul 06 20:00:36.116844 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116770 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "89c834f6-2dc3-496b-8add-58ea612b0e5c" (UID: "89c834f6-2dc3-496b-8add-58ea612b0e5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:36.116844 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116816 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-home" (OuterVolumeSpecName: "home") pod "89c834f6-2dc3-496b-8add-58ea612b0e5c" (UID: "89c834f6-2dc3-496b-8add-58ea612b0e5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:36.116918 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.116857 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89c834f6-2dc3-496b-8add-58ea612b0e5c" (UID: "89c834f6-2dc3-496b-8add-58ea612b0e5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:36.118750 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.118723 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c834f6-2dc3-496b-8add-58ea612b0e5c-kube-api-access-48dn9" (OuterVolumeSpecName: "kube-api-access-48dn9") pod "89c834f6-2dc3-496b-8add-58ea612b0e5c" (UID: "89c834f6-2dc3-496b-8add-58ea612b0e5c"). InnerVolumeSpecName "kube-api-access-48dn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:36.118855 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.118746 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c834f6-2dc3-496b-8add-58ea612b0e5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89c834f6-2dc3-496b-8add-58ea612b0e5c" (UID: "89c834f6-2dc3-496b-8add-58ea612b0e5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:00:36.118855 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.118776 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-dshm" (OuterVolumeSpecName: "dshm") pod "89c834f6-2dc3-496b-8add-58ea612b0e5c" (UID: "89c834f6-2dc3-496b-8add-58ea612b0e5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:36.218019 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.217993 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48dn9\" (UniqueName: \"kubernetes.io/projected/89c834f6-2dc3-496b-8add-58ea612b0e5c-kube-api-access-48dn9\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:36.218019 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.218017 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:36.218165 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.218027 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:36.218165 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.218035 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c834f6-2dc3-496b-8add-58ea612b0e5c-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:36.218165 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.218043 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:36.218165 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.218049 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c834f6-2dc3-496b-8add-58ea612b0e5c-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:36.595972 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.595935 2545 generic.go:356] "Generic (PLEG): container finished" podID="89c834f6-2dc3-496b-8add-58ea612b0e5c" containerID="a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef" exitCode=0 Jul 06 20:00:36.596151 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.595982 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" Jul 06 20:00:36.596151 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.595989 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" event={"ID":"89c834f6-2dc3-496b-8add-58ea612b0e5c","Type":"ContainerDied","Data":"a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef"} Jul 06 20:00:36.596151 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.596027 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8" event={"ID":"89c834f6-2dc3-496b-8add-58ea612b0e5c","Type":"ContainerDied","Data":"a0c3a2bb8e8bbf485eda4712008be2b1d6ba64d78fdd7bbd1a54003257b6dbb6"} Jul 06 20:00:36.596151 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.596049 2545 scope.go:117] "RemoveContainer" containerID="a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef" Jul 06 20:00:36.605407 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.605389 2545 scope.go:117] "RemoveContainer" containerID="a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef" Jul 06 20:00:36.605695 ip-10-0-136-156 kubenswrapper[2545]: E0706 20:00:36.605675 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef\": container with ID starting with a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef not found: ID does not exist" containerID="a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef" Jul 06 20:00:36.605757 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.605706 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef"} err="failed to get container status \"a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef\": rpc error: code = NotFound desc = could not find container \"a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef\": container with ID starting with a5fb7bf2d571dcf11a88783b31c23fb593698a8d6af727351cbf9db762ff32ef not found: ID does not exist" Jul 06 20:00:36.617762 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.617737 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8"] Jul 06 20:00:36.622466 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.622443 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d188f8/test-model-d188f8-kserve-778859bf9f-cjgj8"] Jul 06 20:00:36.727097 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.727040 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" podUID="0f180001-34c8-4979-8d2a-479579571d18" containerName="main" containerID="cri-o://837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2" gracePeriod=45 Jul 06 20:00:36.975381 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:36.975359 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:37.125979 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.125942 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-home\") pod \"0f180001-34c8-4979-8d2a-479579571d18\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " Jul 06 20:00:37.125979 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.125984 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvz5s\" (UniqueName: \"kubernetes.io/projected/0f180001-34c8-4979-8d2a-479579571d18-kube-api-access-wvz5s\") pod \"0f180001-34c8-4979-8d2a-479579571d18\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " Jul 06 20:00:37.126223 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.126025 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f180001-34c8-4979-8d2a-479579571d18-tls-certs\") pod \"0f180001-34c8-4979-8d2a-479579571d18\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " Jul 06 20:00:37.126223 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.126068 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-model-cache\") pod \"0f180001-34c8-4979-8d2a-479579571d18\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " Jul 06 20:00:37.126223 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.126098 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-dshm\") pod \"0f180001-34c8-4979-8d2a-479579571d18\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " Jul 06 20:00:37.126223 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.126141 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-tmp-dir\") pod \"0f180001-34c8-4979-8d2a-479579571d18\" (UID: \"0f180001-34c8-4979-8d2a-479579571d18\") " Jul 06 20:00:37.126462 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.126270 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-home" (OuterVolumeSpecName: "home") pod "0f180001-34c8-4979-8d2a-479579571d18" (UID: "0f180001-34c8-4979-8d2a-479579571d18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:37.126462 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.126376 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f180001-34c8-4979-8d2a-479579571d18" (UID: "0f180001-34c8-4979-8d2a-479579571d18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:37.126462 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.126388 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:37.126462 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.126407 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f180001-34c8-4979-8d2a-479579571d18" (UID: "0f180001-34c8-4979-8d2a-479579571d18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:37.128547 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.128526 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-dshm" (OuterVolumeSpecName: "dshm") pod "0f180001-34c8-4979-8d2a-479579571d18" (UID: "0f180001-34c8-4979-8d2a-479579571d18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:00:37.128547 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.128540 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f180001-34c8-4979-8d2a-479579571d18-kube-api-access-wvz5s" (OuterVolumeSpecName: "kube-api-access-wvz5s") pod "0f180001-34c8-4979-8d2a-479579571d18" (UID: "0f180001-34c8-4979-8d2a-479579571d18"). InnerVolumeSpecName "kube-api-access-wvz5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:00:37.128666 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.128609 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f180001-34c8-4979-8d2a-479579571d18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f180001-34c8-4979-8d2a-479579571d18" (UID: "0f180001-34c8-4979-8d2a-479579571d18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:00:37.227221 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.227134 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvz5s\" (UniqueName: \"kubernetes.io/projected/0f180001-34c8-4979-8d2a-479579571d18-kube-api-access-wvz5s\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:37.227221 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.227170 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f180001-34c8-4979-8d2a-479579571d18-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:37.227221 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.227181 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:37.227221 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.227188 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:37.227221 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.227196 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f180001-34c8-4979-8d2a-479579571d18-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:00:37.600858 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.600762 2545 generic.go:356] "Generic (PLEG): container finished" podID="0f180001-34c8-4979-8d2a-479579571d18" containerID="837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2" exitCode=0 Jul 06 20:00:37.601028 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.600885 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" Jul 06 20:00:37.603150 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.603124 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c834f6-2dc3-496b-8add-58ea612b0e5c" path="/var/lib/kubelet/pods/89c834f6-2dc3-496b-8add-58ea612b0e5c/volumes" Jul 06 20:00:37.603481 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.603465 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" event={"ID":"0f180001-34c8-4979-8d2a-479579571d18","Type":"ContainerDied","Data":"837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2"} Jul 06 20:00:37.603532 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.603492 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh" event={"ID":"0f180001-34c8-4979-8d2a-479579571d18","Type":"ContainerDied","Data":"74cf20be86a52c0474e1d33636f7e721579ad4e0a89ffd92d2be2fce390e523e"} Jul 06 20:00:37.603532 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.603511 2545 scope.go:117] "RemoveContainer" containerID="837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2" Jul 06 20:00:37.612072 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.612039 2545 scope.go:117] "RemoveContainer" containerID="837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2" Jul 06 20:00:37.612356 ip-10-0-136-156 kubenswrapper[2545]: E0706 20:00:37.612334 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2\": container with ID starting with 837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2 not found: ID does not exist" containerID="837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2" Jul 06 20:00:37.612472 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.612363 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2"} err="failed to get container status \"837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2\": rpc error: code = NotFound desc = could not find container \"837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2\": container with ID starting with 837f02c880d1327c2321c9c6a45fdde8cd316c09fea0c2ac7dfdf29e4123d6a2 not found: ID does not exist" Jul 06 20:00:37.624913 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.624888 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh"] Jul 06 20:00:37.629629 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:37.629603 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-62de5f/test-model-62de5f-kserve-6c6777cbb9-k4wxh"] Jul 06 20:00:39.608951 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:39.608912 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f180001-34c8-4979-8d2a-479579571d18" path="/var/lib/kubelet/pods/0f180001-34c8-4979-8d2a-479579571d18/volumes" Jul 06 20:00:59.650370 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:59.650346 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 20:00:59.651948 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:00:59.651929 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 20:01:04.777212 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:04.777183 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:06.101772 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:06.101741 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:06.995085 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:06.995052 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:07.095818 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:07.095786 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:07.793729 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:07.793698 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:07.866234 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:07.866201 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:09.023092 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:09.023056 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:01:12.682225 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:12.682193 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:12.757206 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:12.757170 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:13.113991 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:13.113906 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:13.452308 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:13.452272 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:13.904120 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:13.904088 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:14.315829 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:14.315750 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:14.536920 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:14.536887 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:18.003573 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:18.003539 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:18.110327 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:18.110290 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:18.203091 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:18.203060 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:18.317581 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:18.317508 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:18.786821 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:18.786790 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:18.927818 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:18.927786 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:21.740794 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:21.740762 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:01:22.601347 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:22.601315 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:23.110897 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:23.110868 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:23.617503 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:23.617475 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:24.143031 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:24.143001 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.386366 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:26.386326 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.486943 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:26.486911 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.588217 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:26.588182 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:26.755435 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:26.755341 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:27.100806 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:27.100723 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:27.199939 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:27.199911 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:30.949264 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:30.949236 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:32.525685 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:32.525651 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:01:39.682905 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:39.682873 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:39.783510 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:39.783477 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:39.887134 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:39.887103 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:39.991150 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:39.991067 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:40.215649 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:40.215618 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:43.837764 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:43.837731 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:01:45.947375 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:45.947343 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:46.015346 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:46.015321 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:46.297170 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:46.297096 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:46.521675 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:46.521630 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:46.701021 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:46.700987 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:47.167203 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:47.167174 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:47.273047 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:47.273014 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:47.387604 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:47.387562 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:47.600629 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:47.600545 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:53.786667 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:53.786570 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:53.899926 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:53.899887 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:54.103076 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:54.102999 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:54.502157 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:54.502126 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:54.823651 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:54.823568 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:55.174351 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:55.174320 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:55.442508 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:55.442412 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:57.154845 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:57.154812 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:01:57.919925 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:57.919878 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:58.008080 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:58.008051 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:58.094202 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:58.094177 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:58.594060 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:58.594029 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:01:58.694689 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:01:58.694660 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:02.436269 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:02.436235 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:02.778122 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:02.778037 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:03.096195 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:03.096119 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:03.176947 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:03.176918 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:03.856065 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:03.856021 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:05.619457 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:05.619403 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:06.174284 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:06.174246 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:06.296313 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:06.296290 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:06.372400 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:06.372365 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:06.748900 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:06.748864 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:06.866835 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:06.866806 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:06.966800 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:06.966772 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:07.040190 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:07.040112 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:02:10.231561 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:10.231532 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:13.889803 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:13.889772 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:16.739599 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:16.739563 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:02:19.422534 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:19.422482 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:19.523894 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:19.523865 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:26.535853 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.535817 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7"] Jul 06 20:02:26.536229 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.536078 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c834f6-2dc3-496b-8add-58ea612b0e5c" containerName="main" Jul 06 20:02:26.536229 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.536092 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c834f6-2dc3-496b-8add-58ea612b0e5c" containerName="main" Jul 06 20:02:26.536229 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.536108 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f180001-34c8-4979-8d2a-479579571d18" containerName="main" Jul 06 20:02:26.536229 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.536113 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f180001-34c8-4979-8d2a-479579571d18" containerName="main" Jul 06 20:02:26.536229 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.536180 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f180001-34c8-4979-8d2a-479579571d18" containerName="main" Jul 06 20:02:26.536229 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.536188 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c834f6-2dc3-496b-8add-58ea612b0e5c" containerName="main" Jul 06 20:02:26.538016 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.537999 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.541730 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.541709 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cd90ec\"/\"auth-test-model-cd90ec-kserve-self-signed-certs\"" Jul 06 20:02:26.541873 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.541820 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cd90ec\"/\"openshift-service-ca.crt\"" Jul 06 20:02:26.541873 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.541827 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cd90ec\"/\"default-dockercfg-vjl4v\"" Jul 06 20:02:26.542389 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.542375 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-cd90ec\"/\"kube-root-ca.crt\"" Jul 06 20:02:26.569620 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.569589 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7"] Jul 06 20:02:26.643884 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.643851 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tbk\" (UniqueName: \"kubernetes.io/projected/f0696cbb-02ed-47b9-89b9-fd0903ef553e-kube-api-access-25tbk\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.644006 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.643899 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tls-certs\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.644006 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.643960 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-home\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.644006 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.643978 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-dshm\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.644108 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.644006 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tmp-dir\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.644108 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.644047 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-model-cache\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.663267 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.663234 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:26.744358 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744335 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-model-cache\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.744450 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744371 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25tbk\" (UniqueName: \"kubernetes.io/projected/f0696cbb-02ed-47b9-89b9-fd0903ef553e-kube-api-access-25tbk\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.744450 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744412 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tls-certs\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.744526 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744480 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-home\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.744526 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744507 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-dshm\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.744599 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744530 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tmp-dir\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.744816 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744795 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-model-cache\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.744886 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744827 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-home\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.744886 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.744871 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tmp-dir\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.746835 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.746815 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-dshm\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.746991 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.746975 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tls-certs\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.755957 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.755932 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tbk\" (UniqueName: \"kubernetes.io/projected/f0696cbb-02ed-47b9-89b9-fd0903ef553e-kube-api-access-25tbk\") pod \"auth-test-model-cd90ec-kserve-7895b9df47-brzx7\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.846516 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.846450 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:26.976991 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:26.976962 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7"] Jul 06 20:02:26.978721 ip-10-0-136-156 kubenswrapper[2545]: W0706 20:02:26.978697 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0696cbb_02ed_47b9_89b9_fd0903ef553e.slice/crio-d4ec85c50f697793eeb0dff4a3f651d09d0ed835a1e08bda7400ea7dc9127adf WatchSource:0}: Error finding container d4ec85c50f697793eeb0dff4a3f651d09d0ed835a1e08bda7400ea7dc9127adf: Status 404 returned error can't find the container with id d4ec85c50f697793eeb0dff4a3f651d09d0ed835a1e08bda7400ea7dc9127adf Jul 06 20:02:27.036667 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:27.036639 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" event={"ID":"f0696cbb-02ed-47b9-89b9-fd0903ef553e","Type":"ContainerStarted","Data":"8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580"} Jul 06 20:02:27.036790 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:27.036684 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" event={"ID":"f0696cbb-02ed-47b9-89b9-fd0903ef553e","Type":"ContainerStarted","Data":"d4ec85c50f697793eeb0dff4a3f651d09d0ed835a1e08bda7400ea7dc9127adf"} Jul 06 20:02:27.060171 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:27.060126 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" podStartSLOduration=1.060110046 podStartE2EDuration="1.060110046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:02:27.058031956 +0000 UTC m=+2187.933578642" watchObservedRunningTime="2026-07-06 20:02:27.060110046 +0000 UTC m=+2187.935656727" Jul 06 20:02:27.991840 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:27.991806 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:28.099726 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:28.099699 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:30.001771 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:30.001739 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:02:36.846694 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:36.846657 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:36.847110 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:36.846712 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:36.858679 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:36.858652 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:37.088072 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:37.088046 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:02:47.224080 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:47.224046 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:47.353205 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:47.353173 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:52.207262 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:52.207232 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:52.290248 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:52.290221 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:57.213388 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:57.213344 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:02:57.348444 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:02:57.348403 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:02.212895 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:02.212864 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:02.294947 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:02.294922 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:04.946747 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:04.946714 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6"] Jul 06 20:03:04.949855 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:04.949832 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:04.953101 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:04.953082 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"auth-test-model-04d79c-kserve-self-signed-certs\"" Jul 06 20:03:04.953756 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:04.953740 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"default-dockercfg-ssc5v\"" Jul 06 20:03:04.955589 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:04.955568 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"kube-root-ca.crt\"" Jul 06 20:03:04.955746 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:04.955730 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"openshift-service-ca.crt\"" Jul 06 20:03:04.961014 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:04.960992 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6"] Jul 06 20:03:04.963595 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:04.963577 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:05.000900 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.000876 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tmp-dir\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.000995 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.000917 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktl4\" (UniqueName: \"kubernetes.io/projected/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-kube-api-access-5ktl4\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.000995 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.000947 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-model-cache\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.000995 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.000975 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tls-certs\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.001128 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.001020 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-home\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.001128 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.001038 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-dshm\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102219 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102183 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-home\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102403 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102225 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-dshm\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102403 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102259 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tmp-dir\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102403 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102289 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktl4\" (UniqueName: \"kubernetes.io/projected/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-kube-api-access-5ktl4\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102403 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102329 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-model-cache\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102403 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102358 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tls-certs\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102732 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102571 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-home\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102732 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102700 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tmp-dir\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.102829 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.102736 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-model-cache\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.104627 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.104606 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-dshm\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.104840 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.104820 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tls-certs\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.112554 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.112531 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktl4\" (UniqueName: \"kubernetes.io/projected/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-kube-api-access-5ktl4\") pod \"auth-test-model-04d79c-kserve-5b7db664f5-m4rg6\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.258735 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.258663 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:05.388005 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:05.387979 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6"] Jul 06 20:03:05.389681 ip-10-0-136-156 kubenswrapper[2545]: W0706 20:03:05.389653 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce1c9394_d466_418a_9cfb_5fb46abdb7ab.slice/crio-72e106ab6a8568481a8225ac7d708ff685ec2ccdabe9ae1425a705ca0727d5d4 WatchSource:0}: Error finding container 72e106ab6a8568481a8225ac7d708ff685ec2ccdabe9ae1425a705ca0727d5d4: Status 404 returned error can't find the container with id 72e106ab6a8568481a8225ac7d708ff685ec2ccdabe9ae1425a705ca0727d5d4 Jul 06 20:03:06.193323 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:06.193287 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" event={"ID":"ce1c9394-d466-418a-9cfb-5fb46abdb7ab","Type":"ContainerStarted","Data":"2b10ea939507a203886bb8f69ea08669d76296681ec89f6fc7b2863d35adc0dc"} Jul 06 20:03:06.193817 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:06.193333 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" event={"ID":"ce1c9394-d466-418a-9cfb-5fb46abdb7ab","Type":"ContainerStarted","Data":"72e106ab6a8568481a8225ac7d708ff685ec2ccdabe9ae1425a705ca0727d5d4"} Jul 06 20:03:06.232689 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:06.232632 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" podStartSLOduration=2.232616726 podStartE2EDuration="2.232616726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:03:06.231710069 +0000 UTC m=+2227.107256750" watchObservedRunningTime="2026-07-06 20:03:06.232616726 +0000 UTC m=+2227.108163406" Jul 06 20:03:07.239972 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:07.239936 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:07.412003 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:07.411965 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:09.072769 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:09.072738 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:03:12.273353 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:12.273314 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:12.441851 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:12.441817 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:15.258885 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:15.258855 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:15.258885 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:15.258892 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:15.271201 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:15.271182 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:16.243057 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:16.243028 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:03:17.303015 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:17.302984 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:17.474398 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:17.474360 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:22.244091 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:22.244008 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:22.339877 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:22.339844 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:03:22.347263 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:22.347229 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:25.535275 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:25.535235 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:25.617835 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:25.617805 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:27.232504 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:27.232470 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:27.332732 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:27.332698 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:27.400614 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:27.400585 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:30.366495 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:30.366467 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:30.456978 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:30.456946 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:30.567407 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:30.567368 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:32.265760 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:32.265728 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:37.242041 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:37.242009 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:37.343938 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:37.343910 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:37.443337 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:37.443309 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:38.741796 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:38.741758 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:03:42.251226 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:42.251178 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:42.442012 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:42.441980 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:47.282064 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:47.282032 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:47.380432 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:47.380363 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:47.456484 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:47.456453 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:51.062848 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:51.062812 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:03:52.270073 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:52.270038 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:52.371860 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:52.371829 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:52.456595 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:52.456561 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:57.280624 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:57.280587 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:03:57.384550 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:03:57.384521 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:02.276920 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:02.276884 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:02.382576 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:02.382533 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:02.478411 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:02.478380 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:06.827082 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:06.827050 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:04:07.291762 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:07.291734 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:07.391246 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:07.391214 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:07.457892 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:07.457824 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:12.319452 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:12.319402 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:12.418781 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:12.418747 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:12.492075 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:12.492040 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:17.433198 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:17.433167 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:17.504114 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:17.504084 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:21.571133 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:21.571099 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:04:22.451309 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:22.451280 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:38.928790 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:38.928753 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:04:40.984658 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:40.984625 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:41.063707 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:41.063663 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:41.416293 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:41.416264 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:41.502468 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:41.502433 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:42.870379 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:42.870347 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:04:55.741643 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:55.741557 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:04:59.682797 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:59.682761 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7"] Jul 06 20:04:59.716029 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:04:59.715998 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:00.584232 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:00.584199 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6"] Jul 06 20:05:00.614612 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:00.614586 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:05.834385 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.834347 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz"] Jul 06 20:05:05.837183 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.837167 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.840825 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.840805 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cd90ec\"/\"sub-test-model-cd90ec-kserve-self-signed-certs\"" Jul 06 20:05:05.854982 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.854960 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz"] Jul 06 20:05:05.868895 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.868872 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-model-cache\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.868986 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.868900 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-dshm\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.868986 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.868927 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbrx\" (UniqueName: \"kubernetes.io/projected/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-kube-api-access-tlbrx\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.869072 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.869007 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-home\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.869072 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.869046 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tmp-dir\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.869146 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.869096 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tls-certs\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.883445 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.883399 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:05.970091 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970040 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-model-cache\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.970091 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970098 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-dshm\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.970306 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970123 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbrx\" (UniqueName: \"kubernetes.io/projected/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-kube-api-access-tlbrx\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.970306 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970165 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-home\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.970306 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970202 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tmp-dir\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.970306 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970230 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tls-certs\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.970576 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970555 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-model-cache\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.970638 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970577 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-home\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.970687 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.970656 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tmp-dir\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.972639 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.972620 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-dshm\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.972852 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.972834 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tls-certs\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:05.979792 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:05.979770 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbrx\" (UniqueName: \"kubernetes.io/projected/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-kube-api-access-tlbrx\") pod \"sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:06.145488 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:06.145452 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:06.277658 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:06.277534 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz"] Jul 06 20:05:06.280122 ip-10-0-136-156 kubenswrapper[2545]: W0706 20:05:06.280094 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1619369_d38e_40a6_b9ff_9b0de3bcc9ac.slice/crio-3be28581bb035cf079ef3133003ac0427f6322ff25824d65bc7fd62e1f090806 WatchSource:0}: Error finding container 3be28581bb035cf079ef3133003ac0427f6322ff25824d65bc7fd62e1f090806: Status 404 returned error can't find the container with id 3be28581bb035cf079ef3133003ac0427f6322ff25824d65bc7fd62e1f090806 Jul 06 20:05:06.689163 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:06.689128 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" event={"ID":"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac","Type":"ContainerStarted","Data":"fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d"} Jul 06 20:05:06.689345 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:06.689177 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" event={"ID":"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac","Type":"ContainerStarted","Data":"3be28581bb035cf079ef3133003ac0427f6322ff25824d65bc7fd62e1f090806"} Jul 06 20:05:06.708677 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:06.708630 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" podStartSLOduration=1.7086173740000001 podStartE2EDuration="1.708617374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:05:06.707807897 +0000 UTC m=+2347.583354577" watchObservedRunningTime="2026-07-06 20:05:06.708617374 +0000 UTC m=+2347.584164066" Jul 06 20:05:11.843145 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:11.843112 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:05:14.702400 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:14.702341 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" podUID="f0696cbb-02ed-47b9-89b9-fd0903ef553e" containerName="main" containerID="cri-o://8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580" gracePeriod=45 Jul 06 20:05:14.939077 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:14.939054 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:05:15.043608 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.043532 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tmp-dir\") pod \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " Jul 06 20:05:15.043608 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.043573 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tls-certs\") pod \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " Jul 06 20:05:15.043608 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.043596 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-dshm\") pod \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " Jul 06 20:05:15.043894 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.043724 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-home\") pod \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " Jul 06 20:05:15.043894 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.043754 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25tbk\" (UniqueName: \"kubernetes.io/projected/f0696cbb-02ed-47b9-89b9-fd0903ef553e-kube-api-access-25tbk\") pod \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " Jul 06 20:05:15.043894 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.043778 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-model-cache\") pod \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\" (UID: \"f0696cbb-02ed-47b9-89b9-fd0903ef553e\") " Jul 06 20:05:15.043894 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.043790 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0696cbb-02ed-47b9-89b9-fd0903ef553e" (UID: "f0696cbb-02ed-47b9-89b9-fd0903ef553e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:15.044053 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.044000 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-home" (OuterVolumeSpecName: "home") pod "f0696cbb-02ed-47b9-89b9-fd0903ef553e" (UID: "f0696cbb-02ed-47b9-89b9-fd0903ef553e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:15.044102 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.044049 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0696cbb-02ed-47b9-89b9-fd0903ef553e" (UID: "f0696cbb-02ed-47b9-89b9-fd0903ef553e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:15.044102 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.044075 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:15.044102 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.044088 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:15.046032 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.046005 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-dshm" (OuterVolumeSpecName: "dshm") pod "f0696cbb-02ed-47b9-89b9-fd0903ef553e" (UID: "f0696cbb-02ed-47b9-89b9-fd0903ef553e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:15.046032 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.046018 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0696cbb-02ed-47b9-89b9-fd0903ef553e-kube-api-access-25tbk" (OuterVolumeSpecName: "kube-api-access-25tbk") pod "f0696cbb-02ed-47b9-89b9-fd0903ef553e" (UID: "f0696cbb-02ed-47b9-89b9-fd0903ef553e"). InnerVolumeSpecName "kube-api-access-25tbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:05:15.046187 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.046040 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0696cbb-02ed-47b9-89b9-fd0903ef553e" (UID: "f0696cbb-02ed-47b9-89b9-fd0903ef553e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:05:15.144648 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.144618 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:15.144648 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.144644 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0696cbb-02ed-47b9-89b9-fd0903ef553e-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:15.144648 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.144652 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0696cbb-02ed-47b9-89b9-fd0903ef553e-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:15.144906 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.144660 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tbk\" (UniqueName: \"kubernetes.io/projected/f0696cbb-02ed-47b9-89b9-fd0903ef553e-kube-api-access-25tbk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:15.603798 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.603754 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" podUID="ce1c9394-d466-418a-9cfb-5fb46abdb7ab" containerName="main" containerID="cri-o://2b10ea939507a203886bb8f69ea08669d76296681ec89f6fc7b2863d35adc0dc" gracePeriod=45 Jul 06 20:05:15.727560 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.727529 2545 generic.go:356] "Generic (PLEG): container finished" podID="ce1c9394-d466-418a-9cfb-5fb46abdb7ab" containerID="2b10ea939507a203886bb8f69ea08669d76296681ec89f6fc7b2863d35adc0dc" exitCode=0 Jul 06 20:05:15.727879 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.727600 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" event={"ID":"ce1c9394-d466-418a-9cfb-5fb46abdb7ab","Type":"ContainerDied","Data":"2b10ea939507a203886bb8f69ea08669d76296681ec89f6fc7b2863d35adc0dc"} Jul 06 20:05:15.728911 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.728885 2545 generic.go:356] "Generic (PLEG): container finished" podID="f0696cbb-02ed-47b9-89b9-fd0903ef553e" containerID="8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580" exitCode=0 Jul 06 20:05:15.729021 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.728936 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" Jul 06 20:05:15.729021 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.728930 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" event={"ID":"f0696cbb-02ed-47b9-89b9-fd0903ef553e","Type":"ContainerDied","Data":"8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580"} Jul 06 20:05:15.729109 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.729043 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7" event={"ID":"f0696cbb-02ed-47b9-89b9-fd0903ef553e","Type":"ContainerDied","Data":"d4ec85c50f697793eeb0dff4a3f651d09d0ed835a1e08bda7400ea7dc9127adf"} Jul 06 20:05:15.729109 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.729068 2545 scope.go:117] "RemoveContainer" containerID="8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580" Jul 06 20:05:15.737206 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.737174 2545 scope.go:117] "RemoveContainer" containerID="8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580" Jul 06 20:05:15.737550 ip-10-0-136-156 kubenswrapper[2545]: E0706 20:05:15.737528 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580\": container with ID starting with 8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580 not found: ID does not exist" containerID="8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580" Jul 06 20:05:15.737627 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.737557 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580"} err="failed to get container status \"8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580\": rpc error: code = NotFound desc = could not find container \"8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580\": container with ID starting with 8e6e830f7d690858eb63190bd88946d4135663ba1c6d46bc4aeb1c5009589580 not found: ID does not exist" Jul 06 20:05:15.753361 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.753324 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7"] Jul 06 20:05:15.771793 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.771766 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/auth-test-model-cd90ec-kserve-7895b9df47-brzx7"] Jul 06 20:05:15.852412 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.852393 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:05:15.952488 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952464 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-dshm\") pod \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " Jul 06 20:05:15.952659 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952495 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tls-certs\") pod \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " Jul 06 20:05:15.952659 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952539 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tmp-dir\") pod \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " Jul 06 20:05:15.952659 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952558 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ktl4\" (UniqueName: \"kubernetes.io/projected/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-kube-api-access-5ktl4\") pod \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " Jul 06 20:05:15.952659 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952588 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-model-cache\") pod \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " Jul 06 20:05:15.952659 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952616 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-home\") pod \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\" (UID: \"ce1c9394-d466-418a-9cfb-5fb46abdb7ab\") " Jul 06 20:05:15.952913 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952770 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce1c9394-d466-418a-9cfb-5fb46abdb7ab" (UID: "ce1c9394-d466-418a-9cfb-5fb46abdb7ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:15.952913 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952899 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce1c9394-d466-418a-9cfb-5fb46abdb7ab" (UID: "ce1c9394-d466-418a-9cfb-5fb46abdb7ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:15.952988 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.952915 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-home" (OuterVolumeSpecName: "home") pod "ce1c9394-d466-418a-9cfb-5fb46abdb7ab" (UID: "ce1c9394-d466-418a-9cfb-5fb46abdb7ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:15.954885 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.954864 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-dshm" (OuterVolumeSpecName: "dshm") pod "ce1c9394-d466-418a-9cfb-5fb46abdb7ab" (UID: "ce1c9394-d466-418a-9cfb-5fb46abdb7ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:05:15.954885 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.954873 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce1c9394-d466-418a-9cfb-5fb46abdb7ab" (UID: "ce1c9394-d466-418a-9cfb-5fb46abdb7ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:05:15.954990 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:15.954877 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-kube-api-access-5ktl4" (OuterVolumeSpecName: "kube-api-access-5ktl4") pod "ce1c9394-d466-418a-9cfb-5fb46abdb7ab" (UID: "ce1c9394-d466-418a-9cfb-5fb46abdb7ab"). InnerVolumeSpecName "kube-api-access-5ktl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:05:16.053633 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.053606 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:16.053633 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.053628 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:16.053633 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.053638 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:16.053812 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.053663 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:16.053812 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.053671 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktl4\" (UniqueName: \"kubernetes.io/projected/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-kube-api-access-5ktl4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:16.053812 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.053681 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1c9394-d466-418a-9cfb-5fb46abdb7ab-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:05:16.145778 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.145750 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:16.145911 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.145788 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:16.158564 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.158541 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:16.733709 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.733677 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" Jul 06 20:05:16.734201 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.733676 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6" event={"ID":"ce1c9394-d466-418a-9cfb-5fb46abdb7ab","Type":"ContainerDied","Data":"72e106ab6a8568481a8225ac7d708ff685ec2ccdabe9ae1425a705ca0727d5d4"} Jul 06 20:05:16.734201 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.733801 2545 scope.go:117] "RemoveContainer" containerID="2b10ea939507a203886bb8f69ea08669d76296681ec89f6fc7b2863d35adc0dc" Jul 06 20:05:16.748637 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.748612 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:05:16.758455 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.758434 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6"] Jul 06 20:05:16.766295 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:16.766275 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/auth-test-model-04d79c-kserve-5b7db664f5-m4rg6"] Jul 06 20:05:17.158372 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:17.158346 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:17.603904 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:17.603830 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce1c9394-d466-418a-9cfb-5fb46abdb7ab" path="/var/lib/kubelet/pods/ce1c9394-d466-418a-9cfb-5fb46abdb7ab/volumes" Jul 06 20:05:17.604186 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:17.604174 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0696cbb-02ed-47b9-89b9-fd0903ef553e" path="/var/lib/kubelet/pods/f0696cbb-02ed-47b9-89b9-fd0903ef553e/volumes" Jul 06 20:05:19.815528 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:19.815499 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:19.922673 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:19.922651 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:22.581835 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:22.581799 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:22.671289 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:22.671254 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:26.753269 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.753228 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5"] Jul 06 20:05:26.753749 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.753645 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1c9394-d466-418a-9cfb-5fb46abdb7ab" containerName="main" Jul 06 20:05:26.753749 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.753663 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1c9394-d466-418a-9cfb-5fb46abdb7ab" containerName="main" Jul 06 20:05:26.753749 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.753681 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0696cbb-02ed-47b9-89b9-fd0903ef553e" containerName="main" Jul 06 20:05:26.753749 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.753688 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0696cbb-02ed-47b9-89b9-fd0903ef553e" containerName="main" Jul 06 20:05:26.753966 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.753765 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0696cbb-02ed-47b9-89b9-fd0903ef553e" containerName="main" Jul 06 20:05:26.753966 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.753777 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1c9394-d466-418a-9cfb-5fb46abdb7ab" containerName="main" Jul 06 20:05:26.757876 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.757856 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.760219 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.760196 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"kube-root-ca.crt\"" Jul 06 20:05:26.761108 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.761092 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"openshift-service-ca.crt\"" Jul 06 20:05:26.761163 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.761095 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"default-dockercfg-ssc5v\"" Jul 06 20:05:26.761211 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.761094 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"sub-test-model-04d79c-kserve-self-signed-certs\"" Jul 06 20:05:26.768643 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.768623 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5"] Jul 06 20:05:26.787279 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.787253 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:26.844885 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.844842 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tls-certs\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.844997 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.844893 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-model-cache\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.844997 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.844973 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tmp-dir\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.845086 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.845012 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-home\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.845086 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.845035 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-dshm\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.845086 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.845052 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvff\" (UniqueName: \"kubernetes.io/projected/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-kube-api-access-5xvff\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945520 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945477 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tmp-dir\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945630 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945526 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-home\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945630 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945550 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-dshm\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945630 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945565 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvff\" (UniqueName: \"kubernetes.io/projected/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-kube-api-access-5xvff\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945630 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945589 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tls-certs\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945630 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945608 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-model-cache\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945921 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945898 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tmp-dir\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945980 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945925 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-home\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.945980 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.945969 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-model-cache\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.947926 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.947899 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-dshm\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.948231 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.948213 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tls-certs\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:26.956366 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:26.956344 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvff\" (UniqueName: \"kubernetes.io/projected/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-kube-api-access-5xvff\") pod \"sub-test-model-04d79c-kserve-566576b84f-dfbv5\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:27.066766 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:27.066690 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:27.195952 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:27.195928 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5"] Jul 06 20:05:27.198305 ip-10-0-136-156 kubenswrapper[2545]: W0706 20:05:27.198280 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3094ea96_3933_44f5_9bd8_c2e8ffb14bc0.slice/crio-96ac6c0cab1bedc0be50d033f5ba388598ce59627581b548c7806501b1291663 WatchSource:0}: Error finding container 96ac6c0cab1bedc0be50d033f5ba388598ce59627581b548c7806501b1291663: Status 404 returned error can't find the container with id 96ac6c0cab1bedc0be50d033f5ba388598ce59627581b548c7806501b1291663 Jul 06 20:05:27.780348 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:27.780314 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" event={"ID":"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0","Type":"ContainerStarted","Data":"333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1"} Jul 06 20:05:27.780348 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:27.780353 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" event={"ID":"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0","Type":"ContainerStarted","Data":"96ac6c0cab1bedc0be50d033f5ba388598ce59627581b548c7806501b1291663"} Jul 06 20:05:27.801490 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:27.801345 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" podStartSLOduration=1.8012956930000001 podStartE2EDuration="1.801295693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:05:27.801064277 +0000 UTC m=+2368.676610958" watchObservedRunningTime="2026-07-06 20:05:27.801295693 +0000 UTC m=+2368.676842375" Jul 06 20:05:29.583961 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:29.583924 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:05:32.716205 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:32.716168 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:35.845812 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:35.845785 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:35.946117 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:35.946089 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:37.066968 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:37.066935 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:37.067367 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:37.067214 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:37.079060 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:37.079036 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:37.836784 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:37.836757 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:05:41.968817 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:41.968778 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:42.115179 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:42.115142 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:42.235589 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:42.235506 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:43.555254 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:43.555221 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:05:58.179675 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:58.179645 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz"] Jul 06 20:05:58.221970 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:58.221941 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:59.034685 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:59.034655 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5"] Jul 06 20:05:59.059588 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:59.059558 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:05:59.678262 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:59.678236 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 20:05:59.681185 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:05:59.681164 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7mcmq_0b4dc7c3-5da3-4400-9da7-a00808f289b2/ovn-acl-logging/0.log" Jul 06 20:06:02.338614 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:02.338582 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:06:04.336787 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.336755 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl"] Jul 06 20:06:04.341112 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.341096 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.343503 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.343483 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-cd90ec\"/\"rate-test-model-cd90ec-kserve-self-signed-certs\"" Jul 06 20:06:04.353751 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.353723 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl"] Jul 06 20:06:04.371896 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.371868 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:04.446921 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.446891 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-tmp-dir\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.446921 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.446926 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434d60e6-b477-4adf-bb50-ca014f656ab1-tls-certs\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.447124 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.446948 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-model-cache\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.447124 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.446967 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-dshm\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.447124 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.447012 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjx8w\" (UniqueName: \"kubernetes.io/projected/434d60e6-b477-4adf-bb50-ca014f656ab1-kube-api-access-fjx8w\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.447124 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.447074 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-home\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.548534 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.548499 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-home\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.548715 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.548580 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-tmp-dir\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.548715 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.548612 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434d60e6-b477-4adf-bb50-ca014f656ab1-tls-certs\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.548715 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.548646 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-model-cache\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.548715 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.548673 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-dshm\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.548715 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.548695 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjx8w\" (UniqueName: \"kubernetes.io/projected/434d60e6-b477-4adf-bb50-ca014f656ab1-kube-api-access-fjx8w\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.548990 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.548965 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-home\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.549119 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.549020 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-tmp-dir\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.549119 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.549047 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-model-cache\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.551008 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.550982 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-dshm\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.551247 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.551230 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434d60e6-b477-4adf-bb50-ca014f656ab1-tls-certs\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.559336 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.559313 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjx8w\" (UniqueName: \"kubernetes.io/projected/434d60e6-b477-4adf-bb50-ca014f656ab1-kube-api-access-fjx8w\") pod \"rate-test-model-cd90ec-kserve-97dc4df9-6hldl\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.649504 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.649470 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:04.786066 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.786031 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl"] Jul 06 20:06:04.789800 ip-10-0-136-156 kubenswrapper[2545]: W0706 20:06:04.789774 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod434d60e6_b477_4adf_bb50_ca014f656ab1.slice/crio-78e63b39ad8b60334a4e3e45ab62b7470efcb7f4d234d530b6606dc478575b73 WatchSource:0}: Error finding container 78e63b39ad8b60334a4e3e45ab62b7470efcb7f4d234d530b6606dc478575b73: Status 404 returned error can't find the container with id 78e63b39ad8b60334a4e3e45ab62b7470efcb7f4d234d530b6606dc478575b73 Jul 06 20:06:04.940407 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.940323 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" event={"ID":"434d60e6-b477-4adf-bb50-ca014f656ab1","Type":"ContainerStarted","Data":"d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55"} Jul 06 20:06:04.940407 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.940362 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" event={"ID":"434d60e6-b477-4adf-bb50-ca014f656ab1","Type":"ContainerStarted","Data":"78e63b39ad8b60334a4e3e45ab62b7470efcb7f4d234d530b6606dc478575b73"} Jul 06 20:06:04.979074 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:04.979011 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" podStartSLOduration=0.978996633 podStartE2EDuration="978.996633ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:06:04.978584515 +0000 UTC m=+2405.854131197" watchObservedRunningTime="2026-07-06 20:06:04.978996633 +0000 UTC m=+2405.854543343" Jul 06 20:06:13.199217 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.199170 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" podUID="e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" containerName="main" containerID="cri-o://fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d" gracePeriod=45 Jul 06 20:06:13.448443 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.448400 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:06:13.520823 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.520734 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlbrx\" (UniqueName: \"kubernetes.io/projected/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-kube-api-access-tlbrx\") pod \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " Jul 06 20:06:13.520823 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.520795 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-dshm\") pod \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " Jul 06 20:06:13.520823 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.520819 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tls-certs\") pod \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " Jul 06 20:06:13.521068 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.520879 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-model-cache\") pod \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " Jul 06 20:06:13.521068 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.520927 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tmp-dir\") pod \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " Jul 06 20:06:13.521068 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.520957 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-home\") pod \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\" (UID: \"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac\") " Jul 06 20:06:13.521218 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.521146 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" (UID: "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:13.521310 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.521283 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" (UID: "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:13.521371 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.521307 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-home" (OuterVolumeSpecName: "home") pod "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" (UID: "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:13.523171 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.523146 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" (UID: "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:13.523171 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.523144 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-dshm" (OuterVolumeSpecName: "dshm") pod "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" (UID: "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:13.523308 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.523233 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-kube-api-access-tlbrx" (OuterVolumeSpecName: "kube-api-access-tlbrx") pod "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" (UID: "e1619369-d38e-40a6-b9ff-9b0de3bcc9ac"). InnerVolumeSpecName "kube-api-access-tlbrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:13.621884 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.621846 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:13.621884 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.621876 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:13.621884 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.621889 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbrx\" (UniqueName: \"kubernetes.io/projected/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-kube-api-access-tlbrx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:13.622093 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.621901 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:13.622093 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.621913 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:13.622093 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.621923 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:13.978966 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.978934 2545 generic.go:356] "Generic (PLEG): container finished" podID="e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" containerID="fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d" exitCode=0 Jul 06 20:06:13.978966 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.978967 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" Jul 06 20:06:13.979274 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.978974 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" event={"ID":"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac","Type":"ContainerDied","Data":"fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d"} Jul 06 20:06:13.979274 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.979011 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz" event={"ID":"e1619369-d38e-40a6-b9ff-9b0de3bcc9ac","Type":"ContainerDied","Data":"3be28581bb035cf079ef3133003ac0427f6322ff25824d65bc7fd62e1f090806"} Jul 06 20:06:13.979274 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.979031 2545 scope.go:117] "RemoveContainer" containerID="fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d" Jul 06 20:06:13.987360 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.987331 2545 scope.go:117] "RemoveContainer" containerID="fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d" Jul 06 20:06:13.987655 ip-10-0-136-156 kubenswrapper[2545]: E0706 20:06:13.987633 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d\": container with ID starting with fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d not found: ID does not exist" containerID="fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d" Jul 06 20:06:13.987707 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:13.987671 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d"} err="failed to get container status \"fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d\": rpc error: code = NotFound desc = could not find container \"fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d\": container with ID starting with fcb9390125bc7b11ea9df826778a40b5c937de18727799e85f12b0fd0755da6d not found: ID does not exist" Jul 06 20:06:14.054397 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.054363 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" podUID="3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" containerName="main" containerID="cri-o://333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1" gracePeriod=45 Jul 06 20:06:14.311522 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.311500 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:06:14.428847 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.428813 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-model-cache\") pod \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " Jul 06 20:06:14.429238 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.428857 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tls-certs\") pod \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " Jul 06 20:06:14.429238 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.428886 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xvff\" (UniqueName: \"kubernetes.io/projected/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-kube-api-access-5xvff\") pod \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " Jul 06 20:06:14.429238 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.428932 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-dshm\") pod \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " Jul 06 20:06:14.429238 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.428973 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-home\") pod \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " Jul 06 20:06:14.429238 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.428999 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tmp-dir\") pod \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\" (UID: \"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0\") " Jul 06 20:06:14.429238 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.429121 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" (UID: "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:14.429611 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.429243 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-home" (OuterVolumeSpecName: "home") pod "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" (UID: "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:14.429611 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.429296 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" (UID: "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:14.429611 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.429324 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:14.429611 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.429343 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:14.431295 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.431273 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" (UID: "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:06:14.431295 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.431282 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-dshm" (OuterVolumeSpecName: "dshm") pod "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" (UID: "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:06:14.431451 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.431307 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-kube-api-access-5xvff" (OuterVolumeSpecName: "kube-api-access-5xvff") pod "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" (UID: "3094ea96-3933-44f5-9bd8-c2e8ffb14bc0"). InnerVolumeSpecName "kube-api-access-5xvff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:06:14.530162 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.530120 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:14.530162 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.530154 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:14.530162 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.530163 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xvff\" (UniqueName: \"kubernetes.io/projected/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-kube-api-access-5xvff\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:14.530162 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.530173 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:06:14.650016 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.649982 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:14.650152 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.650037 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:14.662546 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.662514 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:14.988454 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.988355 2545 generic.go:356] "Generic (PLEG): container finished" podID="3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" containerID="333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1" exitCode=0 Jul 06 20:06:14.988454 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.988395 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" Jul 06 20:06:14.988666 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.988448 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" event={"ID":"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0","Type":"ContainerDied","Data":"333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1"} Jul 06 20:06:14.988666 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.988488 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5" event={"ID":"3094ea96-3933-44f5-9bd8-c2e8ffb14bc0","Type":"ContainerDied","Data":"96ac6c0cab1bedc0be50d033f5ba388598ce59627581b548c7806501b1291663"} Jul 06 20:06:14.988666 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.988506 2545 scope.go:117] "RemoveContainer" containerID="333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1" Jul 06 20:06:14.997086 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.997069 2545 scope.go:117] "RemoveContainer" containerID="333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1" Jul 06 20:06:14.997389 ip-10-0-136-156 kubenswrapper[2545]: E0706 20:06:14.997372 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1\": container with ID starting with 333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1 not found: ID does not exist" containerID="333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1" Jul 06 20:06:14.997464 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:14.997397 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1"} err="failed to get container status \"333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1\": rpc error: code = NotFound desc = could not find container \"333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1\": container with ID starting with 333cfd10ae4f139303dc9c37c730c195a93fe22870f8f7b6f7307aa57918fca1 not found: ID does not exist" Jul 06 20:06:15.000795 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:15.000775 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:06:19.054612 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:19.054555 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz"] Jul 06 20:06:19.059658 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:19.059630 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/sub-test-model-cd90ec-kserve-7cc7f87f95-b9rdz"] Jul 06 20:06:19.061249 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:19.061224 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:19.604212 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:19.604179 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" path="/var/lib/kubelet/pods/e1619369-d38e-40a6-b9ff-9b0de3bcc9ac/volumes" Jul 06 20:06:23.576080 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:23.576002 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5"] Jul 06 20:06:25.606514 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:25.606478 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/sub-test-model-04d79c-kserve-566576b84f-dfbv5"] Jul 06 20:06:25.620710 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:25.620676 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:06:27.604190 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:27.604159 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" path="/var/lib/kubelet/pods/3094ea96-3933-44f5-9bd8-c2e8ffb14bc0/volumes" Jul 06 20:06:33.050576 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:06:33.050539 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:07:06.818913 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:06.818884 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:06.919097 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:06.919061 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:08.036873 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.036838 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r"] Jul 06 20:07:08.037241 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.037098 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" containerName="main" Jul 06 20:07:08.037241 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.037109 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" containerName="main" Jul 06 20:07:08.037241 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.037125 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" containerName="main" Jul 06 20:07:08.037241 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.037130 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" containerName="main" Jul 06 20:07:08.037241 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.037178 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="3094ea96-3933-44f5-9bd8-c2e8ffb14bc0" containerName="main" Jul 06 20:07:08.037241 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.037184 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1619369-d38e-40a6-b9ff-9b0de3bcc9ac" containerName="main" Jul 06 20:07:08.039984 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.039966 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.042237 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.042214 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"openshift-service-ca.crt\"" Jul 06 20:07:08.042954 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.042936 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"default-dockercfg-ssc5v\"" Jul 06 20:07:08.043043 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.042968 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"kube-root-ca.crt\"" Jul 06 20:07:08.043043 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.042941 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-04d79c\"/\"rate-test-model-04d79c-kserve-self-signed-certs\"" Jul 06 20:07:08.052300 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.052281 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r"] Jul 06 20:07:08.055972 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.055950 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f88d470-3f52-4677-9a27-39375a89b7ba-tls-certs\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.056074 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.055981 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-dshm\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.056074 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.056005 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p94p\" (UniqueName: \"kubernetes.io/projected/1f88d470-3f52-4677-9a27-39375a89b7ba-kube-api-access-6p94p\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.056074 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.056037 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-model-cache\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.056238 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.056075 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-home\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.056238 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.056119 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-tmp-dir\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.152697 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.152668 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:08.156458 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156412 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-model-cache\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.156572 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156474 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-home\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.156572 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156511 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-tmp-dir\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.156572 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156539 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f88d470-3f52-4677-9a27-39375a89b7ba-tls-certs\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.156740 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156590 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-dshm\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.156740 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156639 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p94p\" (UniqueName: \"kubernetes.io/projected/1f88d470-3f52-4677-9a27-39375a89b7ba-kube-api-access-6p94p\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.156844 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156793 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-model-cache\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.156844 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156821 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-tmp-dir\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.156954 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.156928 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-home\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.158900 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.158879 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-dshm\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.159118 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.159098 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f88d470-3f52-4677-9a27-39375a89b7ba-tls-certs\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.165741 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.165719 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p94p\" (UniqueName: \"kubernetes.io/projected/1f88d470-3f52-4677-9a27-39375a89b7ba-kube-api-access-6p94p\") pod \"rate-test-model-04d79c-kserve-679868c695-kb26r\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.349779 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.349680 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:08.479888 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:08.479854 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r"] Jul 06 20:07:08.482786 ip-10-0-136-156 kubenswrapper[2545]: W0706 20:07:08.482757 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f88d470_3f52_4677_9a27_39375a89b7ba.slice/crio-61e795ef1e5cdb0da430f4de0a8330133f0dec2db7b3299f345e2144b841d6ab WatchSource:0}: Error finding container 61e795ef1e5cdb0da430f4de0a8330133f0dec2db7b3299f345e2144b841d6ab: Status 404 returned error can't find the container with id 61e795ef1e5cdb0da430f4de0a8330133f0dec2db7b3299f345e2144b841d6ab Jul 06 20:07:09.205600 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:09.205559 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" event={"ID":"1f88d470-3f52-4677-9a27-39375a89b7ba","Type":"ContainerStarted","Data":"5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2"} Jul 06 20:07:09.205959 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:09.205610 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" event={"ID":"1f88d470-3f52-4677-9a27-39375a89b7ba","Type":"ContainerStarted","Data":"61e795ef1e5cdb0da430f4de0a8330133f0dec2db7b3299f345e2144b841d6ab"} Jul 06 20:07:09.225802 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:09.225755 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" podStartSLOduration=1.225740284 podStartE2EDuration="1.225740284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:07:09.224619814 +0000 UTC m=+2470.100166495" watchObservedRunningTime="2026-07-06 20:07:09.225740284 +0000 UTC m=+2470.101287005" Jul 06 20:07:11.638881 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:11.638845 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:07:18.350785 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:18.350740 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:18.351186 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:18.350819 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:18.363723 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:18.363695 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:19.258536 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:19.258499 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:07:20.787946 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:20.787916 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:21.326870 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:21.326839 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:29.051198 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:29.051161 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:29.188471 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:29.188442 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:34.631178 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:34.631143 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:07:48.945448 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:48.945399 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:07:55.737659 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:55.737585 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl"] Jul 06 20:07:55.771990 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:55.771961 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:56.654169 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:56.654138 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r"] Jul 06 20:07:56.683981 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:56.683955 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:57.862779 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:57.862746 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:58.317082 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:58.317056 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:58.415568 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:58.415536 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:07:58.613077 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:07:58.613001 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:01.958164 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:01.958130 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:02.537086 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:02.537048 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:08:03.379584 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:03.379548 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:10.757057 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:10.756992 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" podUID="434d60e6-b477-4adf-bb50-ca014f656ab1" containerName="main" containerID="cri-o://d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55" gracePeriod=45 Jul 06 20:08:11.003476 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.003452 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:08:11.079794 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.079713 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-model-cache\") pod \"434d60e6-b477-4adf-bb50-ca014f656ab1\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " Jul 06 20:08:11.079794 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.079765 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-dshm\") pod \"434d60e6-b477-4adf-bb50-ca014f656ab1\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " Jul 06 20:08:11.080013 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.079804 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434d60e6-b477-4adf-bb50-ca014f656ab1-tls-certs\") pod \"434d60e6-b477-4adf-bb50-ca014f656ab1\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " Jul 06 20:08:11.080013 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.079818 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-home\") pod \"434d60e6-b477-4adf-bb50-ca014f656ab1\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " Jul 06 20:08:11.080013 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.079843 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-tmp-dir\") pod \"434d60e6-b477-4adf-bb50-ca014f656ab1\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " Jul 06 20:08:11.080013 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.079876 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjx8w\" (UniqueName: \"kubernetes.io/projected/434d60e6-b477-4adf-bb50-ca014f656ab1-kube-api-access-fjx8w\") pod \"434d60e6-b477-4adf-bb50-ca014f656ab1\" (UID: \"434d60e6-b477-4adf-bb50-ca014f656ab1\") " Jul 06 20:08:11.080230 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.080027 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-model-cache" (OuterVolumeSpecName: "model-cache") pod "434d60e6-b477-4adf-bb50-ca014f656ab1" (UID: "434d60e6-b477-4adf-bb50-ca014f656ab1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:11.080230 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.080063 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-home" (OuterVolumeSpecName: "home") pod "434d60e6-b477-4adf-bb50-ca014f656ab1" (UID: "434d60e6-b477-4adf-bb50-ca014f656ab1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:11.080230 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.080087 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "434d60e6-b477-4adf-bb50-ca014f656ab1" (UID: "434d60e6-b477-4adf-bb50-ca014f656ab1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:11.082160 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.082127 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-dshm" (OuterVolumeSpecName: "dshm") pod "434d60e6-b477-4adf-bb50-ca014f656ab1" (UID: "434d60e6-b477-4adf-bb50-ca014f656ab1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:11.082271 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.082164 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434d60e6-b477-4adf-bb50-ca014f656ab1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "434d60e6-b477-4adf-bb50-ca014f656ab1" (UID: "434d60e6-b477-4adf-bb50-ca014f656ab1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:08:11.082271 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.082178 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434d60e6-b477-4adf-bb50-ca014f656ab1-kube-api-access-fjx8w" (OuterVolumeSpecName: "kube-api-access-fjx8w") pod "434d60e6-b477-4adf-bb50-ca014f656ab1" (UID: "434d60e6-b477-4adf-bb50-ca014f656ab1"). InnerVolumeSpecName "kube-api-access-fjx8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:08:11.181433 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.181387 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:11.181433 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.181435 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjx8w\" (UniqueName: \"kubernetes.io/projected/434d60e6-b477-4adf-bb50-ca014f656ab1-kube-api-access-fjx8w\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:11.181615 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.181445 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:11.181615 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.181454 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:11.181615 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.181462 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434d60e6-b477-4adf-bb50-ca014f656ab1-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:11.181615 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.181469 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434d60e6-b477-4adf-bb50-ca014f656ab1-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:11.450632 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.450603 2545 generic.go:356] "Generic (PLEG): container finished" podID="434d60e6-b477-4adf-bb50-ca014f656ab1" containerID="d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55" exitCode=0 Jul 06 20:08:11.450815 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.450639 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" event={"ID":"434d60e6-b477-4adf-bb50-ca014f656ab1","Type":"ContainerDied","Data":"d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55"} Jul 06 20:08:11.450815 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.450645 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" Jul 06 20:08:11.450815 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.450667 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl" event={"ID":"434d60e6-b477-4adf-bb50-ca014f656ab1","Type":"ContainerDied","Data":"78e63b39ad8b60334a4e3e45ab62b7470efcb7f4d234d530b6606dc478575b73"} Jul 06 20:08:11.450815 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.450683 2545 scope.go:117] "RemoveContainer" containerID="d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55" Jul 06 20:08:11.459977 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.459958 2545 scope.go:117] "RemoveContainer" containerID="d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55" Jul 06 20:08:11.460243 ip-10-0-136-156 kubenswrapper[2545]: E0706 20:08:11.460222 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55\": container with ID starting with d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55 not found: ID does not exist" containerID="d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55" Jul 06 20:08:11.460286 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.460252 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55"} err="failed to get container status \"d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55\": rpc error: code = NotFound desc = could not find container \"d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55\": container with ID starting with d44fb9b5028c2d29f0bc07d8997133d9ff5e06cb0cdfd62216a7646a0ca3ac55 not found: ID does not exist" Jul 06 20:08:11.474901 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.474879 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl"] Jul 06 20:08:11.480396 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.480375 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-cd90ec/rate-test-model-cd90ec-kserve-97dc4df9-6hldl"] Jul 06 20:08:11.603381 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.603348 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="434d60e6-b477-4adf-bb50-ca014f656ab1" path="/var/lib/kubelet/pods/434d60e6-b477-4adf-bb50-ca014f656ab1/volumes" Jul 06 20:08:11.674018 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.673990 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" podUID="1f88d470-3f52-4677-9a27-39375a89b7ba" containerName="main" containerID="cri-o://5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2" gracePeriod=45 Jul 06 20:08:11.917822 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.917797 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:08:11.988215 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988177 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f88d470-3f52-4677-9a27-39375a89b7ba-tls-certs\") pod \"1f88d470-3f52-4677-9a27-39375a89b7ba\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " Jul 06 20:08:11.988384 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988233 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p94p\" (UniqueName: \"kubernetes.io/projected/1f88d470-3f52-4677-9a27-39375a89b7ba-kube-api-access-6p94p\") pod \"1f88d470-3f52-4677-9a27-39375a89b7ba\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " Jul 06 20:08:11.988384 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988266 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-model-cache\") pod \"1f88d470-3f52-4677-9a27-39375a89b7ba\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " Jul 06 20:08:11.988384 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988305 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-dshm\") pod \"1f88d470-3f52-4677-9a27-39375a89b7ba\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " Jul 06 20:08:11.988384 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988323 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-home\") pod \"1f88d470-3f52-4677-9a27-39375a89b7ba\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " Jul 06 20:08:11.988384 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988360 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-tmp-dir\") pod \"1f88d470-3f52-4677-9a27-39375a89b7ba\" (UID: \"1f88d470-3f52-4677-9a27-39375a89b7ba\") " Jul 06 20:08:11.988663 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988558 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f88d470-3f52-4677-9a27-39375a89b7ba" (UID: "1f88d470-3f52-4677-9a27-39375a89b7ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:11.988716 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988680 2545 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:11.988716 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988679 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f88d470-3f52-4677-9a27-39375a89b7ba" (UID: "1f88d470-3f52-4677-9a27-39375a89b7ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:11.988716 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.988692 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-home" (OuterVolumeSpecName: "home") pod "1f88d470-3f52-4677-9a27-39375a89b7ba" (UID: "1f88d470-3f52-4677-9a27-39375a89b7ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:11.990359 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.990338 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f88d470-3f52-4677-9a27-39375a89b7ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f88d470-3f52-4677-9a27-39375a89b7ba" (UID: "1f88d470-3f52-4677-9a27-39375a89b7ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:08:11.990467 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.990448 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-dshm" (OuterVolumeSpecName: "dshm") pod "1f88d470-3f52-4677-9a27-39375a89b7ba" (UID: "1f88d470-3f52-4677-9a27-39375a89b7ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 20:08:11.990517 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:11.990500 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f88d470-3f52-4677-9a27-39375a89b7ba-kube-api-access-6p94p" (OuterVolumeSpecName: "kube-api-access-6p94p") pod "1f88d470-3f52-4677-9a27-39375a89b7ba" (UID: "1f88d470-3f52-4677-9a27-39375a89b7ba"). InnerVolumeSpecName "kube-api-access-6p94p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:08:12.089587 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.089516 2545 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:12.089587 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.089540 2545 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:12.089587 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.089549 2545 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f88d470-3f52-4677-9a27-39375a89b7ba-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:12.089587 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.089558 2545 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f88d470-3f52-4677-9a27-39375a89b7ba-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:12.089587 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.089566 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p94p\" (UniqueName: \"kubernetes.io/projected/1f88d470-3f52-4677-9a27-39375a89b7ba-kube-api-access-6p94p\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Jul 06 20:08:12.455709 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.455670 2545 generic.go:356] "Generic (PLEG): container finished" podID="1f88d470-3f52-4677-9a27-39375a89b7ba" containerID="5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2" exitCode=0 Jul 06 20:08:12.455893 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.455714 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" Jul 06 20:08:12.455893 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.455754 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" event={"ID":"1f88d470-3f52-4677-9a27-39375a89b7ba","Type":"ContainerDied","Data":"5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2"} Jul 06 20:08:12.455893 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.455796 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r" event={"ID":"1f88d470-3f52-4677-9a27-39375a89b7ba","Type":"ContainerDied","Data":"61e795ef1e5cdb0da430f4de0a8330133f0dec2db7b3299f345e2144b841d6ab"} Jul 06 20:08:12.455893 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.455813 2545 scope.go:117] "RemoveContainer" containerID="5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2" Jul 06 20:08:12.464499 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.464481 2545 scope.go:117] "RemoveContainer" containerID="5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2" Jul 06 20:08:12.464722 ip-10-0-136-156 kubenswrapper[2545]: E0706 20:08:12.464707 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2\": container with ID starting with 5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2 not found: ID does not exist" containerID="5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2" Jul 06 20:08:12.464765 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.464731 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2"} err="failed to get container status \"5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2\": rpc error: code = NotFound desc = could not find container \"5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2\": container with ID starting with 5742f9d16ff3529bee663376ed0f8d4932270d618a3234d8a70035eff741daa2 not found: ID does not exist" Jul 06 20:08:12.479564 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.479543 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r"] Jul 06 20:08:12.493085 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:12.493060 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-04d79c/rate-test-model-04d79c-kserve-679868c695-kb26r"] Jul 06 20:08:13.604925 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:13.604890 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f88d470-3f52-4677-9a27-39375a89b7ba" path="/var/lib/kubelet/pods/1f88d470-3f52-4677-9a27-39375a89b7ba/volumes" Jul 06 20:08:35.857245 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:35.857211 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:35.953874 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:35.953842 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:41.423605 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:41.423570 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:41.719054 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:41.718968 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:41.974017 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:41.973940 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:42.288052 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:42.287963 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:42.726225 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:42.726194 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:42.875121 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:42.875088 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:42.983511 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:42.983384 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:43.154473 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:43.154412 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:45.629938 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:45.629905 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6lsqh"] Jul 06 20:08:49.466139 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:49.466109 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:49.553500 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:49.553464 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:49.799403 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:49.799317 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:50.063150 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:50.063075 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:08:50.523216 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:50.523187 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bc667b49-sj47w_f53f4bfd-437d-4650-a445-2f09705909ba/authorino/0.log" Jul 06 20:08:56.728340 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:56.728303 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-6gxfp_57d7db8a-d2d4-45d3-923e-bd8dffb2faa3/copy-manifests/0.log" Jul 06 20:08:56.737078 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:56.737058 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-6gxfp_57d7db8a-d2d4-45d3-923e-bd8dffb2faa3/manager/0.log" Jul 06 20:08:58.605611 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:58.605583 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bc667b49-sj47w_f53f4bfd-437d-4650-a445-2f09705909ba/authorino/0.log" Jul 06 20:08:58.988640 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:58.988618 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-gcftt_8037be85-5de3-46eb-855c-06745c4dc394/kuadrant-console-plugin/0.log" Jul 06 20:08:59.389743 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:08:59.389715 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6lsqh_7c63ff5c-06c6-4341-9e73-92b7e06327ac/limitador/0.log" Jul 06 20:09:00.288540 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:00.288515 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c47f5985b-t46xl_7052a6a3-f7d2-4e06-a39e-2309f5ad66b8/kube-auth-proxy/0.log" Jul 06 20:09:01.528703 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:01.528669 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbm8fpd_4b3b95ae-3a83-4638-8cd8-d62b766bc6fe/main/0.log" Jul 06 20:09:01.782186 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:01.782109 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-9kzfg_972c377b-2b56-4831-9716-0b17049c3894/main/0.log" Jul 06 20:09:06.404864 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:06.404836 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:06.588876 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:06.588851 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:09.575198 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:09.575168 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cs4pr_fdd867de-45fa-4f74-9908-1ff7cb1314e9/global-pull-secret-syncer/0.log" Jul 06 20:09:09.672587 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:09.672562 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" Jul 06 20:09:09.803741 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:09.803716 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-156.ec2.internal_ceedd80492377588ff46c4ba5374d82e/haproxy/0.log" Jul 06 20:09:15.027847 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:15.027807 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bc667b49-sj47w_f53f4bfd-437d-4650-a445-2f09705909ba/authorino/0.log" Jul 06 20:09:15.130101 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:15.130073 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-gcftt_8037be85-5de3-46eb-855c-06745c4dc394/kuadrant-console-plugin/0.log" Jul 06 20:09:15.381828 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:15.381755 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6lsqh_7c63ff5c-06c6-4341-9e73-92b7e06327ac/limitador/0.log" Jul 06 20:09:17.927239 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:17.927207 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlgv9_063be4ab-50c4-4a34-9065-6e09067cefe4/node-exporter/0.log" Jul 06 20:09:17.958390 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:17.958353 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlgv9_063be4ab-50c4-4a34-9065-6e09067cefe4/kube-rbac-proxy/0.log" Jul 06 20:09:17.985233 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:17.985212 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tlgv9_063be4ab-50c4-4a34-9065-6e09067cefe4/init-textfile/0.log" Jul 06 20:09:19.364085 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.364051 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn"] Jul 06 20:09:19.364640 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.364484 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434d60e6-b477-4adf-bb50-ca014f656ab1" containerName="main" Jul 06 20:09:19.364640 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.364502 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="434d60e6-b477-4adf-bb50-ca014f656ab1" containerName="main" Jul 06 20:09:19.364640 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.364531 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f88d470-3f52-4677-9a27-39375a89b7ba" containerName="main" Jul 06 20:09:19.364640 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.364540 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f88d470-3f52-4677-9a27-39375a89b7ba" containerName="main" Jul 06 20:09:19.364640 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.364616 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f88d470-3f52-4677-9a27-39375a89b7ba" containerName="main" Jul 06 20:09:19.364640 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.364626 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="434d60e6-b477-4adf-bb50-ca014f656ab1" containerName="main" Jul 06 20:09:19.367587 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.367567 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.369964 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.369944 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-szkqh\"/\"kube-root-ca.crt\"" Jul 06 20:09:19.370066 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.369946 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-szkqh\"/\"openshift-service-ca.crt\"" Jul 06 20:09:19.370706 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.370689 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-szkqh\"/\"default-dockercfg-bgf4z\"" Jul 06 20:09:19.375121 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.375100 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn"] Jul 06 20:09:19.401262 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.401231 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-proc\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.401445 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.401281 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8f9b\" (UniqueName: \"kubernetes.io/projected/5cd1ab14-22e5-4009-bd68-2008f09b1b65-kube-api-access-b8f9b\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.401445 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.401305 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-sys\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.401445 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.401319 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-podres\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.401445 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.401348 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-lib-modules\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502095 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502058 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8f9b\" (UniqueName: \"kubernetes.io/projected/5cd1ab14-22e5-4009-bd68-2008f09b1b65-kube-api-access-b8f9b\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502261 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502116 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-sys\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502261 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502145 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-podres\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502261 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502188 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-lib-modules\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502261 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502248 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-proc\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502261 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502249 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-sys\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502512 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502300 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-podres\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502512 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502316 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-proc\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.502512 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.502376 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cd1ab14-22e5-4009-bd68-2008f09b1b65-lib-modules\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.511968 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.511937 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8f9b\" (UniqueName: \"kubernetes.io/projected/5cd1ab14-22e5-4009-bd68-2008f09b1b65-kube-api-access-b8f9b\") pod \"perf-node-gather-daemonset-8ljgn\" (UID: \"5cd1ab14-22e5-4009-bd68-2008f09b1b65\") " pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.678162 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.678133 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:19.807696 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.807668 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn"] Jul 06 20:09:19.810332 ip-10-0-136-156 kubenswrapper[2545]: W0706 20:09:19.810302 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5cd1ab14_22e5_4009_bd68_2008f09b1b65.slice/crio-68e13f04371f4331dc8685637c0c8560dfee9f0368de101f3a42b7973f83c85d WatchSource:0}: Error finding container 68e13f04371f4331dc8685637c0c8560dfee9f0368de101f3a42b7973f83c85d: Status 404 returned error can't find the container with id 68e13f04371f4331dc8685637c0c8560dfee9f0368de101f3a42b7973f83c85d Jul 06 20:09:19.932540 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:19.932480 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-4gj67_2bb6c8ee-b9f3-485f-bf40-e4457aec4f1b/networking-console-plugin/0.log" Jul 06 20:09:20.725346 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:20.725265 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" event={"ID":"5cd1ab14-22e5-4009-bd68-2008f09b1b65","Type":"ContainerStarted","Data":"caa1d932a17c9daefebccd9fab494ea29880a20f0d0829c60f7349008d2ddb56"} Jul 06 20:09:20.725346 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:20.725313 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" event={"ID":"5cd1ab14-22e5-4009-bd68-2008f09b1b65","Type":"ContainerStarted","Data":"68e13f04371f4331dc8685637c0c8560dfee9f0368de101f3a42b7973f83c85d"} Jul 06 20:09:20.725814 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:20.725385 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:20.742192 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:20.742151 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" podStartSLOduration=1.742138883 podStartE2EDuration="1.742138883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:09:20.74152833 +0000 UTC m=+2601.617075012" watchObservedRunningTime="2026-07-06 20:09:20.742138883 +0000 UTC m=+2601.617685563" Jul 06 20:09:21.704440 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:21.704389 2545 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9xtwb_f53f1f29-1c7d-473f-b148-fab71b101e48/konnectivity-agent/0.log" limit=1024 Jul 06 20:09:23.156916 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:23.156886 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqdtt_9b877b8c-c234-4d29-aa5f-078c5de3726e/dns/0.log" Jul 06 20:09:23.187562 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:23.187537 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqdtt_9b877b8c-c234-4d29-aa5f-078c5de3726e/kube-rbac-proxy/0.log" Jul 06 20:09:23.271450 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:23.271408 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tcfl6_0dd9923b-0994-4f34-a64b-fb5de561aefd/dns-node-resolver/0.log" Jul 06 20:09:23.950877 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:23.950850 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lz9j9_a211f40a-7213-4ea6-9716-7b8e1a139c9e/node-ca/0.log" Jul 06 20:09:25.275703 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:25.275673 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c47f5985b-t46xl_7052a6a3-f7d2-4e06-a39e-2309f5ad66b8/kube-auth-proxy/0.log" Jul 06 20:09:26.276146 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:26.276115 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d6m7w_5c05a8cd-b758-43e3-9259-0ef57bcd2b6a/serve-healthcheck-canary/0.log" Jul 06 20:09:26.738996 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:26.738971 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-szkqh/perf-node-gather-daemonset-8ljgn" Jul 06 20:09:27.138895 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:27.138869 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nzb5n_5e62983e-ae49-42d3-b0c1-293fee95364f/kube-rbac-proxy/0.log" Jul 06 20:09:27.181676 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:27.181650 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nzb5n_5e62983e-ae49-42d3-b0c1-293fee95364f/exporter/0.log" Jul 06 20:09:27.215075 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:27.215055 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nzb5n_5e62983e-ae49-42d3-b0c1-293fee95364f/extractor/0.log" Jul 06 20:09:29.750114 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:29.750083 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c47f5985b-t46xl_7052a6a3-f7d2-4e06-a39e-2309f5ad66b8/kube-auth-proxy/0.log" Jul 06 20:09:30.324169 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:30.324139 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-6gxfp_57d7db8a-d2d4-45d3-923e-bd8dffb2faa3/manager/0.log" Jul 06 20:09:30.383136 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:30.383113 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586496c87-6gxfp_57d7db8a-d2d4-45d3-923e-bd8dffb2faa3/copy-manifests/0.log" Jul 06 20:09:31.268271 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:31.268242 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c47f5985b-t46xl_7052a6a3-f7d2-4e06-a39e-2309f5ad66b8/kube-auth-proxy/0.log" Jul 06 20:09:32.042192 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:32.042165 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-94489db4c-b5dqb_6c092ad3-3008-4a3e-bf03-963b4d69388a/manager/1.log" Jul 06 20:09:32.042365 ip-10-0-136-156 kubenswrapper[2545]: I0706 20:09:32.042226 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-94489db4c-b5dqb_6c092ad3-3008-4a3e-bf03-963b4d69388a/manager/0.log"