Jul 15 13:06:24.717993 ip-10-0-131-140 systemd[1]: Starting Kubernetes Kubelet... Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.122190 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128087 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128100 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128103 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128107 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128110 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128113 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128116 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:25.216359 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128119 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:25.199342 ip-10-0-131-140 systemd[1]: Started Kubernetes Kubelet. Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128122 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128125 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128127 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128130 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128133 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128135 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128138 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128140 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128144 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128147 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128149 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128152 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128155 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128158 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128161 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128164 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128166 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128168 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128171 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:25.217956 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128173 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128176 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128179 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128181 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128183 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128187 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128192 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128195 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128197 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128203 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128206 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128209 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128212 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128214 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128217 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128219 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128223 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128226 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:25.219035 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128228 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128231 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128233 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128236 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128238 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128241 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128245 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128248 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128251 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128254 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128256 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128259 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128261 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128264 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128266 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128269 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128271 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128274 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128276 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128279 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:25.219901 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128281 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128284 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128287 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128289 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128292 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128294 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128297 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128299 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128302 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128304 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128307 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128309 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128311 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128314 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128318 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128320 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128323 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128325 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128329 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:25.220965 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128332 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128336 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128339 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128342 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128345 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128348 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128351 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128353 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128356 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128358 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128361 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128363 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128366 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128368 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.128371 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130390 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130401 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130405 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130408 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:25.222656 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130411 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130414 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130417 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130421 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130424 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130427 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130429 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130432 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130435 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130437 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130440 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130443 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130447 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130451 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130455 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130458 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130460 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130464 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130466 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:25.223517 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130469 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130472 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130475 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130478 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130480 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130483 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130486 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130488 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130496 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130498 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130502 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130504 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130507 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130509 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130512 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130515 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130517 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130520 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130523 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130526 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:25.224376 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130529 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130531 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130534 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130536 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130539 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130541 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130544 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130546 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130549 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130552 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130554 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130559 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130562 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130565 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130568 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130570 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130573 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130576 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130578 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130581 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:25.225232 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130584 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130586 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130589 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130591 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130593 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130596 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130599 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130602 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130606 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130608 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130611 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130614 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130617 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130620 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130622 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130625 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130627 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130630 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130632 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130635 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:25.226541 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130639 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130643 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130646 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130648 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130655 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130658 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130660 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130663 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130665 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130668 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130670 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130673 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130675 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130678 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130681 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.130683 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131109 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131116 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131120 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131123 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:25.227380 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131126 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131129 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131132 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131135 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131138 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131141 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131144 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131147 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131149 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131152 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131155 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131158 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131161 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131164 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131167 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131169 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131172 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131174 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131177 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:25.228663 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131179 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131182 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131184 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131187 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131189 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131192 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131195 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131199 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131202 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131204 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131208 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131210 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131213 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131215 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131218 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131221 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131224 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131226 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131229 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131232 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:25.229526 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131235 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131237 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131240 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131242 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131245 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131248 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131250 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131253 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131255 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131258 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131260 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131263 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131265 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131268 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131272 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131275 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131278 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131281 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131284 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:25.230997 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131287 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131289 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131292 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131295 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131297 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131300 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131303 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131315 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131318 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131322 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131326 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131330 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131333 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131336 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131339 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131342 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131345 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131348 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131350 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:25.231874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131353 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131356 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131359 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131361 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131364 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131367 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131369 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131372 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131375 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131377 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131380 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131386 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131388 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131391 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131394 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131396 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131399 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.131401 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131467 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131474 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131480 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 13:06:25.233168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131485 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131491 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131494 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131498 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131502 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131505 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131508 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131512 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131515 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131518 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131521 2582 flags.go:64] FLAG: --cgroup-root="" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131524 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131527 2582 flags.go:64] FLAG: --client-ca-file="" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131530 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131534 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131538 2582 flags.go:64] FLAG: --cluster-domain="" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131541 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131544 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131547 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131550 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131554 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131557 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131561 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 13:06:25.234047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131564 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131568 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131571 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131573 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131577 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131580 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131584 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131587 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131590 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131592 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131596 2582 flags.go:64] FLAG: --enable-server="true" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131600 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131604 2582 flags.go:64] FLAG: --event-burst="100" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131607 2582 flags.go:64] FLAG: --event-qps="50" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131610 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131613 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131616 2582 flags.go:64] FLAG: --eviction-hard="" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131619 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131622 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131625 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131628 2582 flags.go:64] FLAG: --eviction-soft="" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131630 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131633 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131637 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131640 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 13:06:25.235346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131643 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131645 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131648 2582 flags.go:64] FLAG: --feature-gates="" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131652 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131655 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131658 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131662 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131665 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131668 2582 flags.go:64] FLAG: --help="false" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131671 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131674 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131677 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131680 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131683 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131686 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131689 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131692 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131699 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131702 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131705 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131708 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131724 2582 flags.go:64] FLAG: --kube-reserved="" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131728 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131731 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 13:06:25.236433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131734 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131737 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131739 2582 flags.go:64] FLAG: --lock-file="" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131742 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131745 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131749 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131759 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131762 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131765 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131768 2582 flags.go:64] FLAG: --logging-format="text" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131771 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131774 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131777 2582 flags.go:64] FLAG: --manifest-url="" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131779 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131786 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131789 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131793 2582 flags.go:64] FLAG: --max-pods="110" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131796 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131799 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131802 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131805 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131808 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131811 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131814 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131821 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 13:06:25.238429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131825 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131828 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131831 2582 flags.go:64] FLAG: --pod-cidr="" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131833 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131840 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131843 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131846 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131849 2582 flags.go:64] FLAG: --port="10250" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131852 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131855 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f0d927925a201a8" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131858 2582 flags.go:64] FLAG: --qos-reserved="" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131861 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131864 2582 flags.go:64] FLAG: --register-node="true" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131866 2582 flags.go:64] FLAG: --register-with-taints="" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131870 2582 flags.go:64] FLAG: --registry-burst="10" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131873 2582 flags.go:64] FLAG: --registry-qps="5" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131876 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131879 2582 flags.go:64] FLAG: --reserved-memory="" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131882 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131885 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131888 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131891 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131895 2582 flags.go:64] FLAG: --runonce="false" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131898 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131901 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 13:06:25.239455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131904 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131907 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131909 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131912 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131915 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131918 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131921 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131925 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131928 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131931 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131934 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131937 2582 flags.go:64] FLAG: --system-cgroups="" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131940 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131945 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131948 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131951 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131955 2582 flags.go:64] FLAG: --tls-min-version="" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131957 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131960 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131963 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131966 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131969 2582 flags.go:64] FLAG: --v="2" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131973 2582 flags.go:64] FLAG: --version="false" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131977 2582 flags.go:64] FLAG: --vmodule="" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131980 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 13:06:25.240580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.131983 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132103 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132106 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132109 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132114 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132117 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132120 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132123 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132126 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132129 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132131 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132134 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132137 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132140 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132143 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132146 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132148 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132151 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132154 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132156 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:25.241682 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132159 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132161 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132163 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132166 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132169 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132171 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132174 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132177 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132179 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132182 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132184 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132189 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132192 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132195 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132197 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132200 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132205 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132207 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132210 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132213 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:25.243191 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132215 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132218 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132220 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132223 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132227 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132230 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132235 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132238 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132241 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132244 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132246 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132249 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132251 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132254 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132257 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132259 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132263 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132266 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132269 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:25.244070 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132272 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132274 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132277 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132279 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132282 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132285 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132287 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132290 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132292 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132295 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132298 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132301 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132303 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132306 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132309 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132311 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132314 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132316 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132319 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132323 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:25.245358 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132326 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132329 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132331 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132334 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132336 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132339 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132341 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132344 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132346 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132349 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132351 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132354 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132357 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132359 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132362 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132364 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132367 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132370 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132373 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:25.246214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132385 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.132387 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.133179 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.140172 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.140188 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.140207 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.140216 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140281 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140285 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140289 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140292 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140295 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140298 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140301 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140304 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:25.247498 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140307 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140310 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140313 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140315 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140317 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140321 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140323 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140326 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140329 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140331 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140334 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140337 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140339 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140342 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140344 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140347 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140350 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140352 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140355 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140357 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:25.248351 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140360 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140363 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140366 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140369 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140371 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140374 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140377 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140379 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140382 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140385 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140387 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140390 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140394 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140398 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140401 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140404 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140407 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140409 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140412 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140415 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:25.249640 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140418 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140421 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140423 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140426 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140428 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140431 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140434 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140438 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140442 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140445 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140448 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140451 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140454 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140456 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140460 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140463 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140465 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140469 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140472 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140475 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:25.250512 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140478 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140481 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140484 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140486 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140489 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140492 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140494 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140497 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140500 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140502 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140505 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140508 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140511 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140513 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140516 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140518 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140520 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140523 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140526 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:25.251988 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140528 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140532 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140535 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140538 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140540 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140543 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140546 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140548 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140551 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140556 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140559 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140562 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.140567 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140729 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:25.252841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140737 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140740 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140743 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140747 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140750 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140753 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140756 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140759 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140761 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140764 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140767 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140769 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140772 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140775 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140777 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140780 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140784 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140787 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140790 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:25.253681 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140793 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140797 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140800 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140803 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140805 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140809 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140812 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140814 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140828 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140831 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140834 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140837 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140839 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140842 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140845 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140847 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140851 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140854 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140857 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140859 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:25.254542 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140862 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140864 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140867 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140870 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140872 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140875 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140878 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140880 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140883 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140885 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140888 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140890 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140893 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140896 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140898 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140901 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140904 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140906 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140908 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140911 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:25.255418 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140915 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140925 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140927 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140930 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140933 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140936 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140938 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140941 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140945 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140947 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140950 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140952 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140955 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140958 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140960 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140963 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140966 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140968 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140971 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140973 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:25.256268 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140976 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140978 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140982 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140984 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140987 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140989 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140992 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140994 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140996 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.140999 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141001 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141004 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141007 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141010 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141018 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141021 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141023 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141026 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:25.257141 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:25.141028 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.141033 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.141151 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.143497 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.144727 2582 server.go:1019] "Starting client certificate rotation" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.144861 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.144904 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.147243 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.148736 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.162822 2582 log.go:25] "Validated CRI v1 runtime API" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.164974 2582 log.go:25] "Validated CRI v1 image API" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.165657 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 13:06:25.257994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.168478 2582 fs.go:135] Filesystem UUIDs: map[192d5468-7239-49d0-970b-dd0a868a9357:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a554c4b1-9342-4a15-b0b3-e8e5463fd0fe:/dev/nvme0n1p3] Jul 15 13:06:25.258641 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.168491 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 13:06:25.258854 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.173966 2582 manager.go:216] Machine: {Timestamp:2026-07-15 13:06:25.172029691 +0000 UTC m=+0.341976035 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100606 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eb73aaa3fa69d3407c00fbf056eb7 SystemUUID:ec2eb73a-aa3f-a69d-3407-c00fbf056eb7 BootID:cc5099e6-d2af-480b-8b17-4e0d51fe7347 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:71:2e:54:cd:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:71:2e:54:cd:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:3f:3b:45:e4:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 13:06:25.258854 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.174066 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 13:06:25.258854 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.174166 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 13:06:25.258854 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.174867 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 13:06:25.258854 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.176863 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 13:06:25.258854 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.176892 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 13:06:25.258854 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.177664 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.177673 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.177680 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.178906 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.180559 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.186697 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.186821 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.186846 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.186860 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.186873 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.186882 2582 kubelet.go:398] "Adding apiserver pod source" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.186890 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.187199 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.187215 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.189021 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191179 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191206 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191354 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191366 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191372 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191377 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191391 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191397 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191874 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191880 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191892 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191899 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191915 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.191923 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.192744 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.192752 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.195583 2582 server.go:1268] "Started kubelet" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.195660 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.195728 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.195776 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.199158 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.205057 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.205212 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-140.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.205314 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 13:06:25.471492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.206206 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.206377 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.207590 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.207599 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.207703 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208202 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208218 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.207423 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-140.ec2.internal.18c27866c5f8e6e7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-140.ec2.internal,UID:ip-10-0-131-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-140.ec2.internal,},FirstTimestamp:2026-07-15 13:06:25.195558631 +0000 UTC m=+0.365504975,LastTimestamp:2026-07-15 13:06:25.195558631 +0000 UTC m=+0.365504975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-140.ec2.internal,}" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208315 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208326 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208345 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208426 2582 factory.go:55] Registering systemd factory Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208441 2582 factory.go:223] Registration of the systemd container factory successfully Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208620 2582 factory.go:153] Registering CRI-O factory Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208634 2582 factory.go:223] Registration of the crio container factory successfully Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208685 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208710 2582 factory.go:103] Registering Raw factory Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.208742 2582 manager.go:1196] Started watching for new ooms in manager Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.209056 2582 manager.go:349] Starting recovery of all containers Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.214152 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.214200 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.221025 2582 manager.go:354] Recovery completed Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.225408 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.227061 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f4479" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.227122 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.227153 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.227164 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.227684 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.227694 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.227710 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.227724 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.228963 2582 policy_none.go:49] "None policy: Start" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.228988 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 13:06:25.474254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.229001 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.229099 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.229120 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.229135 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.230115 2582 policy_none.go:47] "Start" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.235029 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f4479" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.263503 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-140.ec2.internal.18c27866c7dabeb9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-140.ec2.internal,UID:ip-10-0-131-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-140.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-140.ec2.internal,},FirstTimestamp:2026-07-15 13:06:25.227136697 +0000 UTC m=+0.397083046,LastTimestamp:2026-07-15 13:06:25.227136697 +0000 UTC m=+0.397083046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-140.ec2.internal,}" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.299313 2582 manager.go:337] "Starting Device Plugin manager" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.299342 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.299352 2582 server.go:87] "Starting device plugin registration server" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.299518 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.299542 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.299600 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.299673 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.299681 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.300254 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.300290 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.400067 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.401072 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.401102 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.401112 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.401138 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.415008 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.415027 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-140.ec2.internal\": node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.453090 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.454650 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.454684 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.454703 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.454733 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 13:06:25.476091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.456682 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:25.484971 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.484715 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.558030 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal"] Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.558108 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.559052 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.559072 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.559085 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.560459 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.560572 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.560598 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.561183 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.561191 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.561210 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.561212 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.561223 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.561230 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.562321 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.562346 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.562971 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.562997 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.563007 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.581324 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-140.ec2.internal\" not found" node="ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.584798 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-140.ec2.internal\" not found" node="ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.585820 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.610763 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67c309d2e981ec2250495bd92ee46d5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal\" (UID: \"67c309d2e981ec2250495bd92ee46d5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.610785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67c309d2e981ec2250495bd92ee46d5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal\" (UID: \"67c309d2e981ec2250495bd92ee46d5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.610802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6517c9ee24f3128cfced7da2acf57315-config\") pod \"kube-apiserver-proxy-ip-10-0-131-140.ec2.internal\" (UID: \"6517c9ee24f3128cfced7da2acf57315\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.697439 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.687505 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.711636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67c309d2e981ec2250495bd92ee46d5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal\" (UID: \"67c309d2e981ec2250495bd92ee46d5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.711665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67c309d2e981ec2250495bd92ee46d5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal\" (UID: \"67c309d2e981ec2250495bd92ee46d5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.711833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6517c9ee24f3128cfced7da2acf57315-config\") pod \"kube-apiserver-proxy-ip-10-0-131-140.ec2.internal\" (UID: \"6517c9ee24f3128cfced7da2acf57315\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.711871 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6517c9ee24f3128cfced7da2acf57315-config\") pod \"kube-apiserver-proxy-ip-10-0-131-140.ec2.internal\" (UID: \"6517c9ee24f3128cfced7da2acf57315\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.711916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67c309d2e981ec2250495bd92ee46d5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal\" (UID: \"67c309d2e981ec2250495bd92ee46d5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.711941 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67c309d2e981ec2250495bd92ee46d5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal\" (UID: \"67c309d2e981ec2250495bd92ee46d5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.793435 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.889104 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.891491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:25.889165 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:25.894755 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.893580 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:26.084112 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:25.994771 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:26.096253 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.144569 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.144726 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:26.196883 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.238952 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 13:01:26 +0000 UTC" deadline="2028-02-04 14:56:43.199696015 +0000 UTC" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.238977 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13657h50m16.960722504s" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:26.297283 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.338992 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.340026 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.340056 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:26.340994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.340067 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:26.501416 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:26.397959 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-140.ec2.internal\" not found" Jul 15 13:06:26.501416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.401257 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:26.501416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.408002 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" Jul 15 13:06:26.501416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.420768 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 13:06:26.501416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.422523 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" Jul 15 13:06:26.501416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.435947 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 13:06:26.501416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.436761 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:26.518740 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.518718 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:26.610868 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:26.610827 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6517c9ee24f3128cfced7da2acf57315.slice/crio-79272f0a5800df13b9ea28316a5ef095695d2c979f327fd8e2dabb0cd611a2dc WatchSource:0}: Error finding container 79272f0a5800df13b9ea28316a5ef095695d2c979f327fd8e2dabb0cd611a2dc: Status 404 returned error can't find the container with id 79272f0a5800df13b9ea28316a5ef095695d2c979f327fd8e2dabb0cd611a2dc Jul 15 13:06:26.611322 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:26.611298 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c309d2e981ec2250495bd92ee46d5b.slice/crio-29283f273235f62d4170c8ed8eae98f10902fd03f492b446a9e143d441510fb9 WatchSource:0}: Error finding container 29283f273235f62d4170c8ed8eae98f10902fd03f492b446a9e143d441510fb9: Status 404 returned error can't find the container with id 29283f273235f62d4170c8ed8eae98f10902fd03f492b446a9e143d441510fb9 Jul 15 13:06:26.615449 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:26.615430 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:06:27.190159 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.190109 2582 apiserver.go:52] "Watching apiserver" Jul 15 13:06:27.196231 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.196208 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 13:06:27.197240 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.197217 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr","openshift-cluster-node-tuning-operator/tuned-sxdwj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal","openshift-network-operator/iptables-alerter-ng4s4","openshift-ovn-kubernetes/ovnkube-node-zwvqz","kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal","openshift-dns/node-resolver-vjvdt","openshift-image-registry/node-ca-gptn8","openshift-multus/multus-additional-cni-plugins-jfl8z","openshift-multus/multus-n4cml","openshift-multus/network-metrics-daemon-cx6mz","openshift-network-diagnostics/network-check-target-89wts","kube-system/global-pull-secret-syncer-6hjrv","kube-system/konnectivity-agent-9jcjm"] Jul 15 13:06:27.201198 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.201175 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.203322 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.203304 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 13:06:27.203421 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.203362 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nm2tj\"" Jul 15 13:06:27.203421 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.203308 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 13:06:27.203539 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.203435 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.203539 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.203516 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:27.205704 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.205683 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:27.207548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.207514 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 13:06:27.207646 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.207548 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-45nvz\"" Jul 15 13:06:27.207646 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.207539 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 13:06:27.208101 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.208082 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.208644 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.208624 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 13:06:27.209867 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.209846 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:06:27.209867 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.209866 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 13:06:27.210003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.209847 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bdv4c\"" Jul 15 13:06:27.210219 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.210202 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 13:06:27.210501 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.210486 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.212341 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.212326 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 13:06:27.214259 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.213029 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 13:06:27.214259 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.213121 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 13:06:27.214259 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.213250 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 13:06:27.214259 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.213252 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 13:06:27.214259 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.213385 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.214259 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.213404 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 13:06:27.214607 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.214356 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f2z2q\"" Jul 15 13:06:27.216094 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.216011 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 13:06:27.216736 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.216714 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 13:06:27.217065 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.217045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.217757 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.217735 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qst2t\"" Jul 15 13:06:27.217900 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.217884 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 13:06:27.218992 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.218977 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 13:06:27.219167 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219138 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 13:06:27.219311 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219254 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 13:06:27.219311 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219266 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8dvgg\"" Jul 15 13:06:27.219448 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219350 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 13:06:27.219448 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219377 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 13:06:27.219575 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-kubelet\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.219660 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-run-netns\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.219660 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219606 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.219660 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-var-lib-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.219660 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219639 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-etc-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.219904 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219642 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 13:06:27.219904 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219709 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.219904 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovnkube-script-lib\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.219904 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62b177c1-f250-4a0e-9cbb-609fc3ffc741-kubelet-config\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.219904 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-systemd-units\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-env-overrides\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219940 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.220111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d709da95-0380-4aed-ad23-c7d7598e037a-agent-certs\") pod \"konnectivity-agent-9jcjm\" (UID: \"d709da95-0380-4aed-ad23-c7d7598e037a\") " pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:27.220111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.219980 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-host-slash\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.220111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220017 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-systemd\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220083 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57661864-e119-4734-9e4b-f9932acf74e4-hosts-file\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.220111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220108 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-slash\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-cni-netd\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220156 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovnkube-config\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57661864-e119-4734-9e4b-f9932acf74e4-tmp-dir\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220216 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drvn\" (UniqueName: \"kubernetes.io/projected/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-kube-api-access-2drvn\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovn-node-metrics-cert\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d709da95-0380-4aed-ad23-c7d7598e037a-konnectivity-ca\") pod \"konnectivity-agent-9jcjm\" (UID: \"d709da95-0380-4aed-ad23-c7d7598e037a\") " pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220273 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-cni-bin\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220291 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8whg\" (UniqueName: \"kubernetes.io/projected/0ec07d3f-257e-471b-8401-5b2ab083db2d-kube-api-access-l8whg\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220317 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-ovn\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-node-log\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-log-socket\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlprz\" (UniqueName: \"kubernetes.io/projected/57661864-e119-4734-9e4b-f9932acf74e4-kube-api-access-mlprz\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62b177c1-f250-4a0e-9cbb-609fc3ffc741-dbus\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.220493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-iptables-alerter-script\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.221035 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.220435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.221899 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.221881 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 13:06:27.221899 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.221894 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 13:06:27.222041 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.221899 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 13:06:27.222235 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.222220 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.222704 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.222686 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s8bqx\"" Jul 15 13:06:27.224115 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.224097 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k77dx\"" Jul 15 13:06:27.224317 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.224284 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 13:06:27.224317 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.224301 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:06:27.224768 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.224747 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.226506 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.226491 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8j92l\"" Jul 15 13:06:27.226722 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.226708 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 13:06:27.227183 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.227166 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:27.227264 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.227242 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:27.229580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.229561 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:27.229655 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.229617 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:27.243375 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.243357 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7zt6z" Jul 15 13:06:27.254274 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.254255 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7zt6z" Jul 15 13:06:27.309766 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.309744 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 13:06:27.320679 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320658 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-cni-bin\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.320819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-kubernetes\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.320819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-ovn\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.320819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-node-log\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.320819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320756 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-log-socket\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.320819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-log-socket\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.320819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320822 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-node-log\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320816 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-ovn\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320844 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-cni-bin\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62b177c1-f250-4a0e-9cbb-609fc3ffc741-dbus\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-iptables-alerter-script\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-var-lib-kubelet\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-cni-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.320998 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4736cf14-4df8-4a5d-95c1-d026e0d88d42-serviceca\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321022 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-modprobe-d\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321049 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66380cff-7890-4de5-9799-7549ea2954ef-etc-tuned\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62b177c1-f250-4a0e-9cbb-609fc3ffc741-dbus\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-kubelet\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-run-netns\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-var-lib-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321146 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-kubelet\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-etc-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-run-netns\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d709da95-0380-4aed-ad23-c7d7598e037a-agent-certs\") pod \"konnectivity-agent-9jcjm\" (UID: \"d709da95-0380-4aed-ad23-c7d7598e037a\") " pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-registration-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321242 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-var-lib-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-host\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321278 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-etc-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-socket-dir-parent\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-k8s-cni-cncf-io\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-kubelet\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-systemd\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.321731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57661864-e119-4734-9e4b-f9932acf74e4-hosts-file\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321414 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-systemd\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321436 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321449 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321456 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-iptables-alerter-script\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321467 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57661864-e119-4734-9e4b-f9932acf74e4-hosts-file\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndtt\" (UniqueName: \"kubernetes.io/projected/b811d4da-54d9-42e2-b09c-ada3620a5a92-kube-api-access-zndtt\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321533 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-system-cni-dir\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321568 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-conf-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-slash\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321582 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-cni-netd\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.322403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drvn\" (UniqueName: \"kubernetes.io/projected/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-kube-api-access-2drvn\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-kubelet-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321658 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-cnibin\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-slash\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321675 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-lib-modules\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321707 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-cnibin\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-cni-netd\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321756 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovn-node-metrics-cert\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321786 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d709da95-0380-4aed-ad23-c7d7598e037a-konnectivity-ca\") pod \"konnectivity-agent-9jcjm\" (UID: \"d709da95-0380-4aed-ad23-c7d7598e037a\") " pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zp4j\" (UniqueName: \"kubernetes.io/projected/a11f16a4-41fe-4b3f-9c39-4d3161b43505-kube-api-access-9zp4j\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321837 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62h9l\" (UniqueName: \"kubernetes.io/projected/1e2749e6-96d8-4e06-8bdc-0693774149e9-kube-api-access-62h9l\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysctl-d\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-os-release\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321906 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-netns\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.323050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.321965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8whg\" (UniqueName: \"kubernetes.io/projected/0ec07d3f-257e-471b-8401-5b2ab083db2d-kube-api-access-l8whg\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322003 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-sys-fs\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-system-cni-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-cni-bin\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322117 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw67w\" (UniqueName: \"kubernetes.io/projected/ec946d7b-a0ae-471b-90b1-b1735c2ade21-kube-api-access-zw67w\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322151 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprz\" (UniqueName: \"kubernetes.io/projected/57661864-e119-4734-9e4b-f9932acf74e4-kube-api-access-mlprz\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-os-release\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322243 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66380cff-7890-4de5-9799-7549ea2954ef-tmp\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322283 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322328 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-run-openvswitch\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322345 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d709da95-0380-4aed-ad23-c7d7598e037a-konnectivity-ca\") pod \"konnectivity-agent-9jcjm\" (UID: \"d709da95-0380-4aed-ad23-c7d7598e037a\") " pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:27.323608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-cni-multus\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322388 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-hostroot\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-multus-certs\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovnkube-script-lib\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62b177c1-f250-4a0e-9cbb-609fc3ffc741-kubelet-config\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322528 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322550 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-host-slash\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4736cf14-4df8-4a5d-95c1-d026e0d88d42-host\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322591 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysctl-conf\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-systemd\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-host-slash\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62b177c1-f250-4a0e-9cbb-609fc3ffc741-kubelet-config\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-systemd-units\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.324112 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.322741 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322757 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-env-overrides\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.322803 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret podName:62b177c1-f250-4a0e-9cbb-609fc3ffc741 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:27.822781619 +0000 UTC m=+2.992727985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret") pod "global-pull-secret-syncer-6hjrv" (UID: "62b177c1-f250-4a0e-9cbb-609fc3ffc741") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-device-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322874 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ec07d3f-257e-471b-8401-5b2ab083db2d-systemd-units\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-etc-selinux\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-run\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-sys\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.322998 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec946d7b-a0ae-471b-90b1-b1735c2ade21-cni-binary-copy\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-daemon-config\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323053 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovnkube-script-lib\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysconfig\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovnkube-config\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.324589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57661864-e119-4734-9e4b-f9932acf74e4-tmp-dir\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.325024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd92v\" (UniqueName: \"kubernetes.io/projected/4736cf14-4df8-4a5d-95c1-d026e0d88d42-kube-api-access-kd92v\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.325024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.325024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-etc-kubernetes\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.325024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323210 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-socket-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.325024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323221 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-env-overrides\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.325024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323248 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwc7\" (UniqueName: \"kubernetes.io/projected/66380cff-7890-4de5-9799-7549ea2954ef-kube-api-access-hqwc7\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.325024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323411 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57661864-e119-4734-9e4b-f9932acf74e4-tmp-dir\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.325024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.323635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovnkube-config\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.325395 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.325374 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ec07d3f-257e-471b-8401-5b2ab083db2d-ovn-node-metrics-cert\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.325563 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.325546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d709da95-0380-4aed-ad23-c7d7598e037a-agent-certs\") pod \"konnectivity-agent-9jcjm\" (UID: \"d709da95-0380-4aed-ad23-c7d7598e037a\") " pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:27.331227 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.331202 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlprz\" (UniqueName: \"kubernetes.io/projected/57661864-e119-4734-9e4b-f9932acf74e4-kube-api-access-mlprz\") pod \"node-resolver-vjvdt\" (UID: \"57661864-e119-4734-9e4b-f9932acf74e4\") " pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.332247 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.332220 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8whg\" (UniqueName: \"kubernetes.io/projected/0ec07d3f-257e-471b-8401-5b2ab083db2d-kube-api-access-l8whg\") pod \"ovnkube-node-zwvqz\" (UID: \"0ec07d3f-257e-471b-8401-5b2ab083db2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.332353 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.332288 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drvn\" (UniqueName: \"kubernetes.io/projected/8a42b79d-a980-4c1d-9eec-ad71d8a659f8-kube-api-access-2drvn\") pod \"iptables-alerter-ng4s4\" (UID: \"8a42b79d-a980-4c1d-9eec-ad71d8a659f8\") " pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.374810 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.374773 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:27.423545 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-socket-dir-parent\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.423746 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-k8s-cni-cncf-io\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.423746 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-kubelet\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.423746 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423619 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:27.423746 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423636 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-socket-dir-parent\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.423746 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423646 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndtt\" (UniqueName: \"kubernetes.io/projected/b811d4da-54d9-42e2-b09c-ada3620a5a92-kube-api-access-zndtt\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:27.423746 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423751 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-system-cni-dir\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423753 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-kubelet\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423761 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-k8s-cni-cncf-io\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-conf-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423830 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-system-cni-dir\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-kubelet-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.423868 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-cnibin\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-lib-modules\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423902 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-kubelet-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-cnibin\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423869 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-conf-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423929 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-cnibin\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.424184 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.423941 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:27.923919647 +0000 UTC m=+3.093865984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423953 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-cnibin\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.423989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp4j\" (UniqueName: \"kubernetes.io/projected/a11f16a4-41fe-4b3f-9c39-4d3161b43505-kube-api-access-9zp4j\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62h9l\" (UniqueName: \"kubernetes.io/projected/1e2749e6-96d8-4e06-8bdc-0693774149e9-kube-api-access-62h9l\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysctl-d\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-os-release\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424071 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-lib-modules\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424079 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-netns\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424115 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-netns\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-sys-fs\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424134 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-os-release\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-system-cni-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-cni-bin\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw67w\" (UniqueName: \"kubernetes.io/projected/ec946d7b-a0ae-471b-90b1-b1735c2ade21-kube-api-access-zw67w\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-os-release\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424266 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-system-cni-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424266 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysctl-d\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424328 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66380cff-7890-4de5-9799-7549ea2954ef-tmp\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-sys-fs\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424339 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-os-release\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-cni-bin\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-cni-multus\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-hostroot\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-multus-certs\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4736cf14-4df8-4a5d-95c1-d026e0d88d42-host\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysctl-conf\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424622 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-var-lib-cni-multus\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.425866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424640 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-systemd\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424630 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-hostroot\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-host-run-multus-certs\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424701 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-systemd\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424716 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4736cf14-4df8-4a5d-95c1-d026e0d88d42-host\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424737 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-device-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-etc-selinux\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysctl-conf\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-run\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-sys\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-device-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec946d7b-a0ae-471b-90b1-b1735c2ade21-cni-binary-copy\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-daemon-config\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424911 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-etc-selinux\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysconfig\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.426714 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424977 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-run\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.424998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd92v\" (UniqueName: \"kubernetes.io/projected/4736cf14-4df8-4a5d-95c1-d026e0d88d42-kube-api-access-kd92v\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425020 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425055 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e2749e6-96d8-4e06-8bdc-0693774149e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425065 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-sys\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425086 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-sysconfig\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-etc-kubernetes\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425115 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425158 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-etc-kubernetes\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425221 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-socket-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425284 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwc7\" (UniqueName: \"kubernetes.io/projected/66380cff-7890-4de5-9799-7549ea2954ef-kube-api-access-hqwc7\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-kubernetes\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425348 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-var-lib-kubelet\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.427482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425361 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-socket-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425375 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-cni-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4736cf14-4df8-4a5d-95c1-d026e0d88d42-serviceca\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425414 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-var-lib-kubelet\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-modprobe-d\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425448 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec946d7b-a0ae-471b-90b1-b1735c2ade21-cni-binary-copy\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425480 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-cni-dir\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66380cff-7890-4de5-9799-7549ea2954ef-etc-tuned\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425417 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-kubernetes\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-registration-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425555 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e2749e6-96d8-4e06-8bdc-0693774149e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425575 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-etc-modprobe-d\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-host\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425609 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66380cff-7890-4de5-9799-7549ea2954ef-host\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425688 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a11f16a4-41fe-4b3f-9c39-4d3161b43505-registration-dir\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.428202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.425899 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec946d7b-a0ae-471b-90b1-b1735c2ade21-multus-daemon-config\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.429032 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.426042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4736cf14-4df8-4a5d-95c1-d026e0d88d42-serviceca\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.429032 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.427958 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66380cff-7890-4de5-9799-7549ea2954ef-tmp\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.429032 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.428121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66380cff-7890-4de5-9799-7549ea2954ef-etc-tuned\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.443419 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.443351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62h9l\" (UniqueName: \"kubernetes.io/projected/1e2749e6-96d8-4e06-8bdc-0693774149e9-kube-api-access-62h9l\") pod \"multus-additional-cni-plugins-jfl8z\" (UID: \"1e2749e6-96d8-4e06-8bdc-0693774149e9\") " pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.445076 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.444748 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:27.445076 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.444775 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:27.445076 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.444807 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qj62v for pod openshift-network-diagnostics/network-check-target-89wts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:27.445076 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.444895 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v podName:7154e2c8-357d-41e0-83c4-f2cc07b530ad nodeName:}" failed. No retries permitted until 2026-07-15 13:06:27.944875855 +0000 UTC m=+3.114822200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qj62v" (UniqueName: "kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v") pod "network-check-target-89wts" (UID: "7154e2c8-357d-41e0-83c4-f2cc07b530ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:27.446633 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.446208 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndtt\" (UniqueName: \"kubernetes.io/projected/b811d4da-54d9-42e2-b09c-ada3620a5a92-kube-api-access-zndtt\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:27.446633 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.446561 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zp4j\" (UniqueName: \"kubernetes.io/projected/a11f16a4-41fe-4b3f-9c39-4d3161b43505-kube-api-access-9zp4j\") pod \"aws-ebs-csi-driver-node-452wr\" (UID: \"a11f16a4-41fe-4b3f-9c39-4d3161b43505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.446822 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.446635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd92v\" (UniqueName: \"kubernetes.io/projected/4736cf14-4df8-4a5d-95c1-d026e0d88d42-kube-api-access-kd92v\") pod \"node-ca-gptn8\" (UID: \"4736cf14-4df8-4a5d-95c1-d026e0d88d42\") " pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.448540 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.448513 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw67w\" (UniqueName: \"kubernetes.io/projected/ec946d7b-a0ae-471b-90b1-b1735c2ade21-kube-api-access-zw67w\") pod \"multus-n4cml\" (UID: \"ec946d7b-a0ae-471b-90b1-b1735c2ade21\") " pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.449512 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.449463 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwc7\" (UniqueName: \"kubernetes.io/projected/66380cff-7890-4de5-9799-7549ea2954ef-kube-api-access-hqwc7\") pod \"tuned-sxdwj\" (UID: \"66380cff-7890-4de5-9799-7549ea2954ef\") " pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.461345 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.461296 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" event={"ID":"6517c9ee24f3128cfced7da2acf57315","Type":"ContainerStarted","Data":"79272f0a5800df13b9ea28316a5ef095695d2c979f327fd8e2dabb0cd611a2dc"} Jul 15 13:06:27.463318 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.463293 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" event={"ID":"67c309d2e981ec2250495bd92ee46d5b","Type":"ContainerStarted","Data":"29283f273235f62d4170c8ed8eae98f10902fd03f492b446a9e143d441510fb9"} Jul 15 13:06:27.512036 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.511994 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjvdt" Jul 15 13:06:27.520877 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.520849 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:27.524592 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.524565 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57661864_e119_4734_9e4b_f9932acf74e4.slice/crio-f40d4db32f7bb77ad695b4eede10b61f5e3c5c99b67575372ee608b81f8f5ab5 WatchSource:0}: Error finding container f40d4db32f7bb77ad695b4eede10b61f5e3c5c99b67575372ee608b81f8f5ab5: Status 404 returned error can't find the container with id f40d4db32f7bb77ad695b4eede10b61f5e3c5c99b67575372ee608b81f8f5ab5 Jul 15 13:06:27.529560 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.529536 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ng4s4" Jul 15 13:06:27.532441 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.532401 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd709da95_0380_4aed_ad23_c7d7598e037a.slice/crio-f2889f92609b7ed2e6c928bb5edf8c8ac3456503d1cb923d11b5d73daa47723c WatchSource:0}: Error finding container f2889f92609b7ed2e6c928bb5edf8c8ac3456503d1cb923d11b5d73daa47723c: Status 404 returned error can't find the container with id f2889f92609b7ed2e6c928bb5edf8c8ac3456503d1cb923d11b5d73daa47723c Jul 15 13:06:27.533245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.533071 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:27.538751 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.538676 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gptn8" Jul 15 13:06:27.541338 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.541218 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a42b79d_a980_4c1d_9eec_ad71d8a659f8.slice/crio-4c978f487e875f690766328bcf7fcdb7eb8754290c763e6dbdc84f3cb7692135 WatchSource:0}: Error finding container 4c978f487e875f690766328bcf7fcdb7eb8754290c763e6dbdc84f3cb7692135: Status 404 returned error can't find the container with id 4c978f487e875f690766328bcf7fcdb7eb8754290c763e6dbdc84f3cb7692135 Jul 15 13:06:27.544011 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.543990 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" Jul 15 13:06:27.545538 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.545516 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec07d3f_257e_471b_8401_5b2ab083db2d.slice/crio-be82635992bd32aca8892edd11d6401e7f11c64568ee32228b490c6873bb6ce1 WatchSource:0}: Error finding container be82635992bd32aca8892edd11d6401e7f11c64568ee32228b490c6873bb6ce1: Status 404 returned error can't find the container with id be82635992bd32aca8892edd11d6401e7f11c64568ee32228b490c6873bb6ce1 Jul 15 13:06:27.550039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.549890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" Jul 15 13:06:27.552436 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.552410 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4736cf14_4df8_4a5d_95c1_d026e0d88d42.slice/crio-6a06810e7ffc5ed33f8b1b8402531b8fbf33a07170dfe48e73408a9dd177651b WatchSource:0}: Error finding container 6a06810e7ffc5ed33f8b1b8402531b8fbf33a07170dfe48e73408a9dd177651b: Status 404 returned error can't find the container with id 6a06810e7ffc5ed33f8b1b8402531b8fbf33a07170dfe48e73408a9dd177651b Jul 15 13:06:27.555158 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.555029 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" Jul 15 13:06:27.556189 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.556163 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2749e6_96d8_4e06_8bdc_0693774149e9.slice/crio-09a210ae0196ef2ee1e09da369946153f2009268b82dabfb34aadf4d57dcabbc WatchSource:0}: Error finding container 09a210ae0196ef2ee1e09da369946153f2009268b82dabfb34aadf4d57dcabbc: Status 404 returned error can't find the container with id 09a210ae0196ef2ee1e09da369946153f2009268b82dabfb34aadf4d57dcabbc Jul 15 13:06:27.560032 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.559854 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4cml" Jul 15 13:06:27.564024 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.564000 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11f16a4_41fe_4b3f_9c39_4d3161b43505.slice/crio-0788f36df2dbca87f8f5bc1fdb8a6f3c3ac9c133e8bae0b597802a1503597b99 WatchSource:0}: Error finding container 0788f36df2dbca87f8f5bc1fdb8a6f3c3ac9c133e8bae0b597802a1503597b99: Status 404 returned error can't find the container with id 0788f36df2dbca87f8f5bc1fdb8a6f3c3ac9c133e8bae0b597802a1503597b99 Jul 15 13:06:27.570339 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.570305 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66380cff_7890_4de5_9799_7549ea2954ef.slice/crio-75a4005734c27b6674955c8a94f899ba2d77954e57b036a51937541c9486dc56 WatchSource:0}: Error finding container 75a4005734c27b6674955c8a94f899ba2d77954e57b036a51937541c9486dc56: Status 404 returned error can't find the container with id 75a4005734c27b6674955c8a94f899ba2d77954e57b036a51937541c9486dc56 Jul 15 13:06:27.575947 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:06:27.575923 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec946d7b_a0ae_471b_90b1_b1735c2ade21.slice/crio-d4255b22f8a1f9361288ab102e1be0b83c9c637b5f6a984b2c70edb3435fc2f5 WatchSource:0}: Error finding container d4255b22f8a1f9361288ab102e1be0b83c9c637b5f6a984b2c70edb3435fc2f5: Status 404 returned error can't find the container with id d4255b22f8a1f9361288ab102e1be0b83c9c637b5f6a984b2c70edb3435fc2f5 Jul 15 13:06:27.828227 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.828174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:27.828404 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.828323 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:27.828474 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.828406 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret podName:62b177c1-f250-4a0e-9cbb-609fc3ffc741 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:28.828380348 +0000 UTC m=+3.998326695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret") pod "global-pull-secret-syncer-6hjrv" (UID: "62b177c1-f250-4a0e-9cbb-609fc3ffc741") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:27.929045 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:27.929000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:27.929241 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.929175 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:27.929307 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:27.929276 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:28.92923833 +0000 UTC m=+4.099184685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:28.029481 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.029438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:28.029717 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.029693 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:28.029827 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.029744 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:28.029827 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.029762 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qj62v for pod openshift-network-diagnostics/network-check-target-89wts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:28.030014 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.029831 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v podName:7154e2c8-357d-41e0-83c4-f2cc07b530ad nodeName:}" failed. No retries permitted until 2026-07-15 13:06:29.029815335 +0000 UTC m=+4.199761671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj62v" (UniqueName: "kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v") pod "network-check-target-89wts" (UID: "7154e2c8-357d-41e0-83c4-f2cc07b530ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:28.258134 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.258056 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 13:01:27 +0000 UTC" deadline="2028-02-09 09:00:31.784596562 +0000 UTC" Jul 15 13:06:28.258134 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.258133 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13771h54m3.526468857s" Jul 15 13:06:28.454935 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.454900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:28.455538 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.455028 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:28.468849 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.468764 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" event={"ID":"a11f16a4-41fe-4b3f-9c39-4d3161b43505","Type":"ContainerStarted","Data":"0788f36df2dbca87f8f5bc1fdb8a6f3c3ac9c133e8bae0b597802a1503597b99"} Jul 15 13:06:28.471464 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.471433 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" event={"ID":"1e2749e6-96d8-4e06-8bdc-0693774149e9","Type":"ContainerStarted","Data":"09a210ae0196ef2ee1e09da369946153f2009268b82dabfb34aadf4d57dcabbc"} Jul 15 13:06:28.475959 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.475918 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gptn8" event={"ID":"4736cf14-4df8-4a5d-95c1-d026e0d88d42","Type":"ContainerStarted","Data":"6a06810e7ffc5ed33f8b1b8402531b8fbf33a07170dfe48e73408a9dd177651b"} Jul 15 13:06:28.478734 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.478695 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"be82635992bd32aca8892edd11d6401e7f11c64568ee32228b490c6873bb6ce1"} Jul 15 13:06:28.481093 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.481068 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9jcjm" event={"ID":"d709da95-0380-4aed-ad23-c7d7598e037a","Type":"ContainerStarted","Data":"f2889f92609b7ed2e6c928bb5edf8c8ac3456503d1cb923d11b5d73daa47723c"} Jul 15 13:06:28.486281 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.486248 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjvdt" event={"ID":"57661864-e119-4734-9e4b-f9932acf74e4","Type":"ContainerStarted","Data":"f40d4db32f7bb77ad695b4eede10b61f5e3c5c99b67575372ee608b81f8f5ab5"} Jul 15 13:06:28.491060 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.490993 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4cml" event={"ID":"ec946d7b-a0ae-471b-90b1-b1735c2ade21","Type":"ContainerStarted","Data":"d4255b22f8a1f9361288ab102e1be0b83c9c637b5f6a984b2c70edb3435fc2f5"} Jul 15 13:06:28.497114 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.497056 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" event={"ID":"66380cff-7890-4de5-9799-7549ea2954ef","Type":"ContainerStarted","Data":"75a4005734c27b6674955c8a94f899ba2d77954e57b036a51937541c9486dc56"} Jul 15 13:06:28.520614 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.520547 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ng4s4" event={"ID":"8a42b79d-a980-4c1d-9eec-ad71d8a659f8","Type":"ContainerStarted","Data":"4c978f487e875f690766328bcf7fcdb7eb8754290c763e6dbdc84f3cb7692135"} Jul 15 13:06:28.836890 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.836803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:28.837102 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.836950 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:28.837102 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.837018 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret podName:62b177c1-f250-4a0e-9cbb-609fc3ffc741 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:30.836997692 +0000 UTC m=+6.006944040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret") pod "global-pull-secret-syncer-6hjrv" (UID: "62b177c1-f250-4a0e-9cbb-609fc3ffc741") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:28.937948 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:28.937912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:28.938188 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.938168 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:28.938273 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:28.938238 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:30.938218963 +0000 UTC m=+6.108165314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:29.039079 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:29.038351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:29.039079 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:29.038578 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:29.039079 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:29.038603 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:29.039079 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:29.038618 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qj62v for pod openshift-network-diagnostics/network-check-target-89wts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:29.039079 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:29.038691 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v podName:7154e2c8-357d-41e0-83c4-f2cc07b530ad nodeName:}" failed. No retries permitted until 2026-07-15 13:06:31.038657504 +0000 UTC m=+6.208603841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj62v" (UniqueName: "kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v") pod "network-check-target-89wts" (UID: "7154e2c8-357d-41e0-83c4-f2cc07b530ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:29.258812 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:29.258767 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 13:01:27 +0000 UTC" deadline="2028-04-01 19:44:32.951284474 +0000 UTC" Jul 15 13:06:29.258812 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:29.258806 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15030h38m3.692483621s" Jul 15 13:06:29.458115 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:29.458084 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:29.459055 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:29.458799 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:29.466234 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:29.465980 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:29.466234 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:29.466153 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:29.530376 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:29.529398 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" event={"ID":"6517c9ee24f3128cfced7da2acf57315","Type":"ContainerStarted","Data":"0af913f9e2a3029159d91f65e7d22425c1fb0c938a259882912fd37ded7928f5"} Jul 15 13:06:30.455473 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:30.454961 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:30.455473 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:30.455097 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:30.854048 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:30.853298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:30.854048 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:30.853513 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:30.854048 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:30.853577 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret podName:62b177c1-f250-4a0e-9cbb-609fc3ffc741 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:34.853556529 +0000 UTC m=+10.023502869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret") pod "global-pull-secret-syncer-6hjrv" (UID: "62b177c1-f250-4a0e-9cbb-609fc3ffc741") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:30.955086 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:30.954462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:30.955086 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:30.954632 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:30.955086 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:30.954702 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:34.954681389 +0000 UTC m=+10.124627737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:31.055701 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:31.055664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:31.055955 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:31.055936 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:31.056065 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:31.055963 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:31.056065 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:31.055977 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qj62v for pod openshift-network-diagnostics/network-check-target-89wts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:31.056196 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:31.056181 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v podName:7154e2c8-357d-41e0-83c4-f2cc07b530ad nodeName:}" failed. No retries permitted until 2026-07-15 13:06:35.056156436 +0000 UTC m=+10.226102787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj62v" (UniqueName: "kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v") pod "network-check-target-89wts" (UID: "7154e2c8-357d-41e0-83c4-f2cc07b530ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:31.457338 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:31.455464 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:31.457338 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:31.455490 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:31.457338 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:31.455629 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:31.457338 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:31.455792 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:32.455259 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:32.455224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:32.455668 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:32.455366 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:33.455651 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:33.455617 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:33.456152 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:33.455780 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:33.456152 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:33.455845 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:33.456152 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:33.455934 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:34.455503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:34.455332 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:34.455503 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:34.455453 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:34.892351 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:34.892228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:34.892528 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:34.892379 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:34.892528 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:34.892452 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret podName:62b177c1-f250-4a0e-9cbb-609fc3ffc741 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:42.89243132 +0000 UTC m=+18.062377660 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret") pod "global-pull-secret-syncer-6hjrv" (UID: "62b177c1-f250-4a0e-9cbb-609fc3ffc741") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:34.994034 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:34.993384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:34.994034 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:34.993602 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:34.994034 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:34.993680 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:42.993658285 +0000 UTC m=+18.163604621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:35.094804 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:35.094768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:35.094977 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:35.094929 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:35.094977 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:35.094951 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:35.094977 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:35.094963 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qj62v for pod openshift-network-diagnostics/network-check-target-89wts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:35.095151 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:35.095021 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v podName:7154e2c8-357d-41e0-83c4-f2cc07b530ad nodeName:}" failed. No retries permitted until 2026-07-15 13:06:43.09500268 +0000 UTC m=+18.264949031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj62v" (UniqueName: "kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v") pod "network-check-target-89wts" (UID: "7154e2c8-357d-41e0-83c4-f2cc07b530ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:35.459466 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:35.459430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:35.459994 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:35.459558 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:35.459994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:35.459931 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:35.460121 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:35.460068 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:36.455357 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:36.455324 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:36.455573 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:36.455455 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:37.455982 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:37.455947 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:37.456437 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:37.456083 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:37.456548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:37.456526 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:37.456674 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:37.456626 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:38.455148 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:38.455115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:38.455652 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:38.455616 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:39.455945 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:39.455915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:39.456392 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:39.456047 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:39.456392 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:39.456104 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:39.456392 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:39.456224 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:40.455217 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:40.455179 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:40.455389 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:40.455296 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:41.455624 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:41.455581 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:41.456070 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:41.455747 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:41.456070 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:41.455790 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:41.456070 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:41.455906 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:42.454964 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:42.454932 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:42.455160 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:42.455062 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:42.957700 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:42.957667 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:42.958139 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:42.957867 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:42.958139 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:42.957942 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret podName:62b177c1-f250-4a0e-9cbb-609fc3ffc741 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:58.957920244 +0000 UTC m=+34.127866594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret") pod "global-pull-secret-syncer-6hjrv" (UID: "62b177c1-f250-4a0e-9cbb-609fc3ffc741") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:43.058825 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:43.058799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:43.059007 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:43.058949 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:43.059054 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:43.059012 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:59.058993645 +0000 UTC m=+34.228939978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:43.159086 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:43.159060 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:43.159195 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:43.159182 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:43.159246 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:43.159212 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:43.159246 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:43.159221 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qj62v for pod openshift-network-diagnostics/network-check-target-89wts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:43.159309 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:43.159262 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v podName:7154e2c8-357d-41e0-83c4-f2cc07b530ad nodeName:}" failed. No retries permitted until 2026-07-15 13:06:59.159247786 +0000 UTC m=+34.329194118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj62v" (UniqueName: "kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v") pod "network-check-target-89wts" (UID: "7154e2c8-357d-41e0-83c4-f2cc07b530ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:43.458593 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:43.458562 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:43.458765 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:43.458595 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:43.458765 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:43.458738 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:43.458887 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:43.458800 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:44.455444 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:44.455414 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:44.455824 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:44.455514 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:45.455648 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:45.455612 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:45.456112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:45.455661 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:45.456112 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:45.455747 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:45.456112 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:45.455818 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:46.455517 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:46.455488 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:46.455672 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:46.455588 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:47.455635 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:47.455600 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:47.455798 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:47.455694 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:47.455798 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:47.455744 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:47.456077 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:47.455834 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:48.455898 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.455754 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:48.456497 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:48.455968 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:48.577140 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.577037 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4cml" event={"ID":"ec946d7b-a0ae-471b-90b1-b1735c2ade21","Type":"ContainerStarted","Data":"33bf4a7b2ea4b9491a397a4f7c7b6da0907ec980ea58a99ba84c6dcb81f44ce8"} Jul 15 13:06:48.578817 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.578776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" event={"ID":"66380cff-7890-4de5-9799-7549ea2954ef","Type":"ContainerStarted","Data":"3b1ef2da975cc1a6a0fe317b1890d87f072ce92ed9e987f4b290ee5de5408ebf"} Jul 15 13:06:48.580504 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.580477 2582 generic.go:356] "Generic (PLEG): container finished" podID="67c309d2e981ec2250495bd92ee46d5b" containerID="23de6ba0ae823a45a16ba522e7f19649380a0928f67cf4e696710863e2ef0d00" exitCode=0 Jul 15 13:06:48.580612 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.580540 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" event={"ID":"67c309d2e981ec2250495bd92ee46d5b","Type":"ContainerDied","Data":"23de6ba0ae823a45a16ba522e7f19649380a0928f67cf4e696710863e2ef0d00"} Jul 15 13:06:48.582298 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.582264 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" event={"ID":"a11f16a4-41fe-4b3f-9c39-4d3161b43505","Type":"ContainerStarted","Data":"2f5f283c38e2a35d697d4e0967c19122f4981fc59e0ba73f006ba2bbc365754c"} Jul 15 13:06:48.583971 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.583938 2582 generic.go:356] "Generic (PLEG): container finished" podID="1e2749e6-96d8-4e06-8bdc-0693774149e9" containerID="671fb7a6af39345421b363f2c7ee44c209205da6b8b0e87a0890e50ea9b9ef63" exitCode=0 Jul 15 13:06:48.584089 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.584015 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" event={"ID":"1e2749e6-96d8-4e06-8bdc-0693774149e9","Type":"ContainerDied","Data":"671fb7a6af39345421b363f2c7ee44c209205da6b8b0e87a0890e50ea9b9ef63"} Jul 15 13:06:48.585816 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.585757 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gptn8" event={"ID":"4736cf14-4df8-4a5d-95c1-d026e0d88d42","Type":"ContainerStarted","Data":"01cecaad5453aa7c1341c81057737b2f79311c3ecd16525b703e7cc730c52922"} Jul 15 13:06:48.589254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.589237 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:06:48.589664 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.589646 2582 generic.go:356] "Generic (PLEG): container finished" podID="0ec07d3f-257e-471b-8401-5b2ab083db2d" containerID="233e59dc402940d7ac0822ed868a117687ec3c136c654ed45334d1d1d54cf7f2" exitCode=1 Jul 15 13:06:48.589772 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.589707 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"797231ab73d48b6df9085c45fce3385aaf079f559c4c1cdd5692d1b87bf2bd93"} Jul 15 13:06:48.589859 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.589841 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"645df0c53b3c9062064d8036c2037d8a4cfcf45f5c95eaea9f6b1a7a8410171b"} Jul 15 13:06:48.589932 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.589902 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"8b4db0bc418918d72ba50bdafbe42b67f45203272f9101b2941f83a1fd88ae79"} Jul 15 13:06:48.589994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.589953 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"1ea25d3a8e8087ccb2d90a6f882abe9fe97b948c3fbe467384640ad2aac6e8f9"} Jul 15 13:06:48.590046 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.590001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerDied","Data":"233e59dc402940d7ac0822ed868a117687ec3c136c654ed45334d1d1d54cf7f2"} Jul 15 13:06:48.590097 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.590089 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"168da77253f14d14c24e8887fabe14a4b2bbb2e9a29ee76ecdadd0b5b375c634"} Jul 15 13:06:48.591452 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.591333 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9jcjm" event={"ID":"d709da95-0380-4aed-ad23-c7d7598e037a","Type":"ContainerStarted","Data":"c960dbb949f57833a86db37495fe22d7fee09bb259b4da9c20d0d138ddb46c04"} Jul 15 13:06:48.593447 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.593423 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjvdt" event={"ID":"57661864-e119-4734-9e4b-f9932acf74e4","Type":"ContainerStarted","Data":"b054dd89e690dffcd6b1845b5f5abc30beea2b773145179a081648c7be5479ab"} Jul 15 13:06:48.594551 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.594500 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-140.ec2.internal" podStartSLOduration=21.594466581 podStartE2EDuration="21.594466581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:29.549906768 +0000 UTC m=+4.719853126" watchObservedRunningTime="2026-07-15 13:06:48.594466581 +0000 UTC m=+23.764412927" Jul 15 13:06:48.594862 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.594839 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n4cml" podStartSLOduration=2.410732104 podStartE2EDuration="22.594827599s" totalImagesPullingTime="20.184095495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:48.594272618 +0000 UTC m=+23.764218972" watchObservedRunningTime="2026-07-15 13:06:48.594827599 +0000 UTC m=+23.764773954" Jul 15 13:06:48.608232 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.608195 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vjvdt" podStartSLOduration=2.415247555 podStartE2EDuration="22.608183022s" totalImagesPullingTime="20.192935467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:48.608012975 +0000 UTC m=+23.777959329" watchObservedRunningTime="2026-07-15 13:06:48.608183022 +0000 UTC m=+23.778129375" Jul 15 13:06:48.637051 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.637009 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gptn8" podStartSLOduration=2.487681848 podStartE2EDuration="22.636996604s" totalImagesPullingTime="20.149314756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:48.636822858 +0000 UTC m=+23.806769211" watchObservedRunningTime="2026-07-15 13:06:48.636996604 +0000 UTC m=+23.806942959" Jul 15 13:06:48.637353 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.637325 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9jcjm" podStartSLOduration=7.07644897 podStartE2EDuration="22.63731912s" totalImagesPullingTime="15.56087015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:48.622709971 +0000 UTC m=+23.792656343" watchObservedRunningTime="2026-07-15 13:06:48.63731912 +0000 UTC m=+23.807265473" Jul 15 13:06:48.673243 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:48.673201 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sxdwj" podStartSLOduration=2.49601003 podStartE2EDuration="22.673186564s" totalImagesPullingTime="20.177176534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:48.673168957 +0000 UTC m=+23.843115310" watchObservedRunningTime="2026-07-15 13:06:48.673186564 +0000 UTC m=+23.843132917" Jul 15 13:06:49.391042 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:49.391014 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 13:06:49.455460 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:49.455285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:49.455460 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:49.455320 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:49.455657 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:49.455527 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:49.455657 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:49.455574 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:49.596937 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:49.596828 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ng4s4" event={"ID":"8a42b79d-a980-4c1d-9eec-ad71d8a659f8","Type":"ContainerStarted","Data":"72696913847a3136e0dbe3350aa92f78d1a20a4828f684c09b040ebf8aca55e1"} Jul 15 13:06:49.598846 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:49.598818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" event={"ID":"a11f16a4-41fe-4b3f-9c39-4d3161b43505","Type":"ContainerStarted","Data":"faec4ee770e25573a280ef0987723f4b1d9a92c9b1098dc186afc789246f9280"} Jul 15 13:06:49.626209 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:49.626173 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ng4s4" podStartSLOduration=3.421170382 podStartE2EDuration="23.626145428s" totalImagesPullingTime="20.204975046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:49.611524226 +0000 UTC m=+24.781470579" watchObservedRunningTime="2026-07-15 13:06:49.626145428 +0000 UTC m=+24.796091780" Jul 15 13:06:50.313838 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:50.313751 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T13:06:49.391037011Z","UUID":"9e8d9d6c-7d63-4079-94f4-be3edfcfb59d","Handler":null,"Name":"","Endpoint":""} Jul 15 13:06:50.315379 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:50.315355 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 13:06:50.315558 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:50.315396 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 13:06:50.455282 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:50.455244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:50.455424 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:50.455369 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:50.603104 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:50.603071 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" event={"ID":"67c309d2e981ec2250495bd92ee46d5b","Type":"ContainerStarted","Data":"0fbb09dfbd091190e43927f29ac625971362a5e1dcdd9cdb39e13d4710022555"} Jul 15 13:06:50.618905 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:50.618857 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-140.ec2.internal" podStartSLOduration=23.618837821 podStartE2EDuration="23.618837821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:50.618324155 +0000 UTC m=+25.788270510" watchObservedRunningTime="2026-07-15 13:06:50.618837821 +0000 UTC m=+25.788784176" Jul 15 13:06:51.455164 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.454989 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:51.455363 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.455037 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:51.455363 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:51.455281 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:51.455363 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:51.455313 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:51.594958 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.594922 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:51.595525 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.595502 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:51.607421 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.607390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" event={"ID":"a11f16a4-41fe-4b3f-9c39-4d3161b43505","Type":"ContainerStarted","Data":"76dc49e269dc64a266d48269eb67104ff21d27695c43ebf870634b2202656821"} Jul 15 13:06:51.610725 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.610694 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:06:51.611177 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.611154 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"69eb6006d9a3184da2a29cc179b66ab7fde9950ce83fd77e8c21c86c8a97ad42"} Jul 15 13:06:51.611324 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.611304 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:51.611759 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.611707 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9jcjm" Jul 15 13:06:51.652278 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:51.652238 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-452wr" podStartSLOduration=3.233861308 podStartE2EDuration="25.652225664s" totalImagesPullingTime="22.418364356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:51.652132118 +0000 UTC m=+26.822078473" watchObservedRunningTime="2026-07-15 13:06:51.652225664 +0000 UTC m=+26.822172049" Jul 15 13:06:52.455007 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:52.454976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:52.455173 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:52.455103 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:53.455607 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:53.455444 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:53.456253 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:53.455448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:53.456253 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:53.455685 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:53.456253 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:53.455761 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:53.616960 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:53.616930 2582 generic.go:356] "Generic (PLEG): container finished" podID="1e2749e6-96d8-4e06-8bdc-0693774149e9" containerID="bde8445b0bc38a40a4c455e53294ab1426c568e4d72dcff98c1530cc722c578e" exitCode=0 Jul 15 13:06:53.617116 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:53.617006 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" event={"ID":"1e2749e6-96d8-4e06-8bdc-0693774149e9","Type":"ContainerDied","Data":"bde8445b0bc38a40a4c455e53294ab1426c568e4d72dcff98c1530cc722c578e"} Jul 15 13:06:53.620426 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:53.620404 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:06:53.620840 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:53.620819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"d23bedd7b030329aa4f1f4572fff92973adb2b8e7f880663e7dc084dacbabb76"} Jul 15 13:06:53.621302 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:53.621286 2582 scope.go:117] "RemoveContainer" containerID="233e59dc402940d7ac0822ed868a117687ec3c136c654ed45334d1d1d54cf7f2" Jul 15 13:06:54.455518 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.455490 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:54.455767 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:54.455625 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:54.626945 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.626920 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:06:54.627353 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.627324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" event={"ID":"0ec07d3f-257e-471b-8401-5b2ab083db2d","Type":"ContainerStarted","Data":"8ec687f0459156aa4b9e7268ad2895eb1e7540f4eb470083d833ce1a2cdcca8f"} Jul 15 13:06:54.627438 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.627408 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 15 13:06:54.627533 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.627517 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:54.627608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.627581 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:54.647665 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.647641 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:54.648564 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.648547 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:54.704908 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.704867 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" podStartSLOduration=8.413069012 podStartE2EDuration="28.704849345s" totalImagesPullingTime="20.291780333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:54.67119889 +0000 UTC m=+29.841145227" watchObservedRunningTime="2026-07-15 13:06:54.704849345 +0000 UTC m=+29.874795698" Jul 15 13:06:54.826043 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.826014 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89wts"] Jul 15 13:06:54.826157 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.826118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:54.826234 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:54.826192 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:54.829380 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.829358 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6hjrv"] Jul 15 13:06:54.829494 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.829465 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:54.829576 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:54.829559 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:54.830244 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.830223 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cx6mz"] Jul 15 13:06:54.830337 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:54.830323 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:54.830442 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:54.830422 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:55.631487 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:55.631451 2582 generic.go:356] "Generic (PLEG): container finished" podID="1e2749e6-96d8-4e06-8bdc-0693774149e9" containerID="03dc6b2b07fbc5376db6638258ef2d5fa3de0d3dff7a689b4ef78aaf12097369" exitCode=0 Jul 15 13:06:55.631922 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:55.631529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" event={"ID":"1e2749e6-96d8-4e06-8bdc-0693774149e9","Type":"ContainerDied","Data":"03dc6b2b07fbc5376db6638258ef2d5fa3de0d3dff7a689b4ef78aaf12097369"} Jul 15 13:06:55.631922 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:55.631750 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 15 13:06:56.166242 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:56.166216 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:06:56.455316 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:56.455288 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:56.455471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:56.455375 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:56.455471 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:56.455379 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:56.455471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:56.455404 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:56.455574 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:56.455495 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:56.455574 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:56.455564 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:57.637350 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:57.637185 2582 generic.go:356] "Generic (PLEG): container finished" podID="1e2749e6-96d8-4e06-8bdc-0693774149e9" containerID="cf204d3615ae71636211aec6fbc602e1a257e8991b00f08a8505edfbd300ed08" exitCode=0 Jul 15 13:06:57.637832 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:57.637274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" event={"ID":"1e2749e6-96d8-4e06-8bdc-0693774149e9","Type":"ContainerDied","Data":"cf204d3615ae71636211aec6fbc602e1a257e8991b00f08a8505edfbd300ed08"} Jul 15 13:06:57.652169 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:57.652114 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" podUID="0ec07d3f-257e-471b-8401-5b2ab083db2d" containerName="ovnkube-controller" probeResult="failure" output="" Jul 15 13:06:58.455398 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:58.455362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:58.455590 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:58.455362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:58.455590 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:58.455476 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:06:58.455590 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:58.455544 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:06:58.455590 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:58.455363 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:58.455831 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:58.455639 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:06:58.981343 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:58.981301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:06:58.981815 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:58.981463 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:58.981815 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:58.981537 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret podName:62b177c1-f250-4a0e-9cbb-609fc3ffc741 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:30.98151749 +0000 UTC m=+66.151463836 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret") pod "global-pull-secret-syncer-6hjrv" (UID: "62b177c1-f250-4a0e-9cbb-609fc3ffc741") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:59.082663 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:59.082627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:06:59.082844 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:59.082809 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:59.082906 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:59.082873 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:31.082857662 +0000 UTC m=+66.252803997 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:59.183432 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:06:59.183397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:06:59.183608 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:59.183530 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:59.183608 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:59.183552 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:59.183608 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:59.183563 2582 projected.go:196] Error preparing data for projected volume kube-api-access-qj62v for pod openshift-network-diagnostics/network-check-target-89wts: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:59.183785 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:06:59.183611 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v podName:7154e2c8-357d-41e0-83c4-f2cc07b530ad nodeName:}" failed. No retries permitted until 2026-07-15 13:07:31.183592642 +0000 UTC m=+66.353538974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qj62v" (UniqueName: "kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v") pod "network-check-target-89wts" (UID: "7154e2c8-357d-41e0-83c4-f2cc07b530ad") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:07:00.455794 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:00.455763 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:07:00.456311 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:00.455770 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:07:00.456311 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:00.455897 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:07:00.456311 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:00.456002 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hjrv" podUID="62b177c1-f250-4a0e-9cbb-609fc3ffc741" Jul 15 13:07:00.456311 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:00.455778 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:07:00.456311 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:00.456123 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89wts" podUID="7154e2c8-357d-41e0-83c4-f2cc07b530ad" Jul 15 13:07:01.661762 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.661663 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-140.ec2.internal" event="NodeReady" Jul 15 13:07:01.662145 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.661814 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 13:07:01.694573 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.694545 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b"] Jul 15 13:07:01.696091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.696076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.698200 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.698177 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956"] Jul 15 13:07:01.699503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.699475 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 13:07:01.699503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.699495 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 13:07:01.699702 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.699523 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 13:07:01.699702 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.699531 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 13:07:01.699856 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.699840 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf"] Jul 15 13:07:01.700384 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.700353 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:07:01.702388 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.702267 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ppmm7\"" Jul 15 13:07:01.702835 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.702815 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 15 13:07:01.704137 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.703424 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77d7b747c9-fcj99"] Jul 15 13:07:01.704137 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.703663 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.705768 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.705696 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 13:07:01.705768 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.705736 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 13:07:01.706034 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.705813 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 13:07:01.706079 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.706037 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 13:07:01.706312 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.706296 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.708065 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.708046 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 13:07:01.708174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.708104 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 13:07:01.708174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.708148 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j4np7\"" Jul 15 13:07:01.708405 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.708387 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 13:07:01.710947 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.710882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b"] Jul 15 13:07:01.714457 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.714438 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 13:07:01.716673 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.716635 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956"] Jul 15 13:07:01.717387 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.717367 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mjlh5"] Jul 15 13:07:01.719281 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.719265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:01.721136 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.721118 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 13:07:01.721229 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.721178 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-txt24\"" Jul 15 13:07:01.721229 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.721213 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 13:07:01.723197 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.723154 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ghmn2"] Jul 15 13:07:01.725021 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.725003 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf"] Jul 15 13:07:01.725098 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.725028 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d7b747c9-fcj99"] Jul 15 13:07:01.725098 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.725085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghmn2" Jul 15 13:07:01.726808 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.726791 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 13:07:01.726898 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.726812 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 13:07:01.727057 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.727043 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 13:07:01.727157 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.727141 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-htl88\"" Jul 15 13:07:01.730045 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.730026 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mjlh5"] Jul 15 13:07:01.737726 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.737683 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghmn2"] Jul 15 13:07:01.804157 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.804282 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-registry-certificates\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.804282 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804185 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-ca\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.804282 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.804282 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804249 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.804282 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804275 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvzg\" (UniqueName: \"kubernetes.io/projected/b709f9e1-85a0-4d82-adc8-17ee27958e0d-kube-api-access-kbvzg\") pod \"managed-serviceaccount-addon-agent-7cb7bd6b76-sf956\" (UID: \"b709f9e1-85a0-4d82-adc8-17ee27958e0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:07:01.804471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8qj\" (UniqueName: \"kubernetes.io/projected/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-kube-api-access-hq8qj\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.804471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dac0d3ea-32ad-4873-a99d-340a8e421fb6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.804471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42g6p\" (UniqueName: \"kubernetes.io/projected/dac0d3ea-32ad-4873-a99d-340a8e421fb6-kube-api-access-42g6p\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.804471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804419 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-trusted-ca\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.804471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-installation-pull-secrets\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.804471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9gx\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-kube-api-access-cj9gx\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.804691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804474 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b709f9e1-85a0-4d82-adc8-17ee27958e0d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cb7bd6b76-sf956\" (UID: \"b709f9e1-85a0-4d82-adc8-17ee27958e0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:07:01.804691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0657627-2d08-41e9-9326-0f7fc596f470-ca-trust-extracted\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.804691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-bound-sa-token\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.804691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804532 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac0d3ea-32ad-4873-a99d-340a8e421fb6-tmp\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.804691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-hub\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.804691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.804691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.804660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-image-registry-private-configuration\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.905283 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905258 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.905380 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905296 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jx7\" (UniqueName: \"kubernetes.io/projected/429fe006-ddc8-454f-b468-9a6d3f553856-kube-api-access-27jx7\") pod \"ingress-canary-ghmn2\" (UID: \"429fe006-ddc8-454f-b468-9a6d3f553856\") " pod="openshift-ingress-canary/ingress-canary-ghmn2" Jul 15 13:07:01.905380 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905319 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:01.905380 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905344 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xhx\" (UniqueName: \"kubernetes.io/projected/e2e9cb69-d4ea-4457-9b22-29286c5cc772-kube-api-access-g4xhx\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:01.905380 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905371 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429fe006-ddc8-454f-b468-9a6d3f553856-cert\") pod \"ingress-canary-ghmn2\" (UID: \"429fe006-ddc8-454f-b468-9a6d3f553856\") " pod="openshift-ingress-canary/ingress-canary-ghmn2" Jul 15 13:07:01.905567 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-image-registry-private-configuration\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.905567 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.905567 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-registry-certificates\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.905567 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-ca\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.905567 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905520 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.905567 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905568 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvzg\" (UniqueName: \"kubernetes.io/projected/b709f9e1-85a0-4d82-adc8-17ee27958e0d-kube-api-access-kbvzg\") pod \"managed-serviceaccount-addon-agent-7cb7bd6b76-sf956\" (UID: \"b709f9e1-85a0-4d82-adc8-17ee27958e0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:01.905584 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905593 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e9cb69-d4ea-4457-9b22-29286c5cc772-config-volume\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:01.905602 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e9cb69-d4ea-4457-9b22-29286c5cc772-tmp-dir\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8qj\" (UniqueName: \"kubernetes.io/projected/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-kube-api-access-hq8qj\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:01.905671 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:02.405651143 +0000 UTC m=+37.575597491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905700 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dac0d3ea-32ad-4873-a99d-340a8e421fb6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42g6p\" (UniqueName: \"kubernetes.io/projected/dac0d3ea-32ad-4873-a99d-340a8e421fb6-kube-api-access-42g6p\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-trusted-ca\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.905829 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-installation-pull-secrets\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.906297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9gx\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-kube-api-access-cj9gx\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.906297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b709f9e1-85a0-4d82-adc8-17ee27958e0d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cb7bd6b76-sf956\" (UID: \"b709f9e1-85a0-4d82-adc8-17ee27958e0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:07:01.906297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0657627-2d08-41e9-9326-0f7fc596f470-ca-trust-extracted\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.906297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-bound-sa-token\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.906297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac0d3ea-32ad-4873-a99d-340a8e421fb6-tmp\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.906297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.905956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-hub\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.908145 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.908116 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-registry-certificates\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.908578 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.908547 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0657627-2d08-41e9-9326-0f7fc596f470-ca-trust-extracted\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.909771 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.909389 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.910000 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.909887 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac0d3ea-32ad-4873-a99d-340a8e421fb6-tmp\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.910483 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.910458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-hub\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.910628 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.910600 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.911100 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.911080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-ca\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.911255 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.911236 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.911682 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.911660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-trusted-ca\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.911803 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.911680 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-image-registry-private-configuration\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.911853 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.911806 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-installation-pull-secrets\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.912285 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.912260 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dac0d3ea-32ad-4873-a99d-340a8e421fb6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.912872 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.912851 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b709f9e1-85a0-4d82-adc8-17ee27958e0d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cb7bd6b76-sf956\" (UID: \"b709f9e1-85a0-4d82-adc8-17ee27958e0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:07:01.916608 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.916587 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9gx\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-kube-api-access-cj9gx\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.916958 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.916932 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8qj\" (UniqueName: \"kubernetes.io/projected/a59d4b16-44a0-41ba-89ee-ac8ef6d991ef-kube-api-access-hq8qj\") pod \"cluster-proxy-proxy-agent-7c566796b9-6cmrf\" (UID: \"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:01.917706 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.917684 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-bound-sa-token\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:01.918478 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.918457 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42g6p\" (UniqueName: \"kubernetes.io/projected/dac0d3ea-32ad-4873-a99d-340a8e421fb6-kube-api-access-42g6p\") pod \"klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b\" (UID: \"dac0d3ea-32ad-4873-a99d-340a8e421fb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:01.918588 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:01.918570 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvzg\" (UniqueName: \"kubernetes.io/projected/b709f9e1-85a0-4d82-adc8-17ee27958e0d-kube-api-access-kbvzg\") pod \"managed-serviceaccount-addon-agent-7cb7bd6b76-sf956\" (UID: \"b709f9e1-85a0-4d82-adc8-17ee27958e0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:07:02.006544 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.006521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jx7\" (UniqueName: \"kubernetes.io/projected/429fe006-ddc8-454f-b468-9a6d3f553856-kube-api-access-27jx7\") pod \"ingress-canary-ghmn2\" (UID: \"429fe006-ddc8-454f-b468-9a6d3f553856\") " pod="openshift-ingress-canary/ingress-canary-ghmn2" Jul 15 13:07:02.006629 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.006548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:02.006629 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.006565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xhx\" (UniqueName: \"kubernetes.io/projected/e2e9cb69-d4ea-4457-9b22-29286c5cc772-kube-api-access-g4xhx\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:02.006629 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.006588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429fe006-ddc8-454f-b468-9a6d3f553856-cert\") pod \"ingress-canary-ghmn2\" (UID: \"429fe006-ddc8-454f-b468-9a6d3f553856\") " pod="openshift-ingress-canary/ingress-canary-ghmn2" Jul 15 13:07:02.006788 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:02.006651 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:02.006788 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.006678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e9cb69-d4ea-4457-9b22-29286c5cc772-config-volume\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:02.006788 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.006700 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e9cb69-d4ea-4457-9b22-29286c5cc772-tmp-dir\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:02.006788 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:02.006741 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:02.506705295 +0000 UTC m=+37.676651631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:07:02.007063 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.007042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e9cb69-d4ea-4457-9b22-29286c5cc772-tmp-dir\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:02.007249 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.007231 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e9cb69-d4ea-4457-9b22-29286c5cc772-config-volume\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:02.008635 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.008618 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/429fe006-ddc8-454f-b468-9a6d3f553856-cert\") pod \"ingress-canary-ghmn2\" (UID: \"429fe006-ddc8-454f-b468-9a6d3f553856\") " pod="openshift-ingress-canary/ingress-canary-ghmn2" Jul 15 13:07:02.011900 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.011880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:02.014082 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.014061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xhx\" (UniqueName: \"kubernetes.io/projected/e2e9cb69-d4ea-4457-9b22-29286c5cc772-kube-api-access-g4xhx\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:02.014137 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.014084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jx7\" (UniqueName: \"kubernetes.io/projected/429fe006-ddc8-454f-b468-9a6d3f553856-kube-api-access-27jx7\") pod \"ingress-canary-ghmn2\" (UID: \"429fe006-ddc8-454f-b468-9a6d3f553856\") " pod="openshift-ingress-canary/ingress-canary-ghmn2" Jul 15 13:07:02.029169 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.029151 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:07:02.037402 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.037380 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:07:02.073199 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.072843 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghmn2" Jul 15 13:07:02.176661 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.176531 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956"] Jul 15 13:07:02.181478 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:07:02.181439 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb709f9e1_85a0_4d82_adc8_17ee27958e0d.slice/crio-4ca6a8f08de386291e203ae78e71f3580067645e65afda204eded490e75e4ace WatchSource:0}: Error finding container 4ca6a8f08de386291e203ae78e71f3580067645e65afda204eded490e75e4ace: Status 404 returned error can't find the container with id 4ca6a8f08de386291e203ae78e71f3580067645e65afda204eded490e75e4ace Jul 15 13:07:02.186634 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.186600 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b"] Jul 15 13:07:02.189019 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:07:02.188997 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac0d3ea_32ad_4873_a99d_340a8e421fb6.slice/crio-ae7640d0bee568072c87115b07161944390dac53ba436da783ea8ea4d0362769 WatchSource:0}: Error finding container ae7640d0bee568072c87115b07161944390dac53ba436da783ea8ea4d0362769: Status 404 returned error can't find the container with id ae7640d0bee568072c87115b07161944390dac53ba436da783ea8ea4d0362769 Jul 15 13:07:02.191625 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.191572 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf"] Jul 15 13:07:02.194922 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:07:02.194893 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda59d4b16_44a0_41ba_89ee_ac8ef6d991ef.slice/crio-03c1822f96da6e0d604165fad9cf5926aa5a997861881fcda04ca0511f92cb00 WatchSource:0}: Error finding container 03c1822f96da6e0d604165fad9cf5926aa5a997861881fcda04ca0511f92cb00: Status 404 returned error can't find the container with id 03c1822f96da6e0d604165fad9cf5926aa5a997861881fcda04ca0511f92cb00 Jul 15 13:07:02.224872 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.224849 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghmn2"] Jul 15 13:07:02.227615 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:07:02.227594 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429fe006_ddc8_454f_b468_9a6d3f553856.slice/crio-4a4bb95636126634105678822021350872a0d0f79695c31d75a282cac96daa3e WatchSource:0}: Error finding container 4a4bb95636126634105678822021350872a0d0f79695c31d75a282cac96daa3e: Status 404 returned error can't find the container with id 4a4bb95636126634105678822021350872a0d0f79695c31d75a282cac96daa3e Jul 15 13:07:02.410816 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.410795 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:02.410958 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:02.410941 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:07:02.411014 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:02.410962 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:07:02.411014 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:02.411010 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:03.410992608 +0000 UTC m=+38.580938951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:07:02.455631 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.455602 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:07:02.455737 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.455611 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:07:02.456000 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.455611 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:07:02.461102 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.461083 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 13:07:02.461181 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.461112 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 13:07:02.461231 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.461189 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nr6v2\"" Jul 15 13:07:02.461900 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.461883 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 13:07:02.462658 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.462642 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 13:07:02.463100 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.463083 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8tgt2\"" Jul 15 13:07:02.511792 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.511771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:02.511917 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:02.511903 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:02.511971 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:02.511962 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:03.511949593 +0000 UTC m=+38.681895929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:07:02.654871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.654834 2582 generic.go:356] "Generic (PLEG): container finished" podID="1e2749e6-96d8-4e06-8bdc-0693774149e9" containerID="940e3efa188ca82552b131c77f67ff1063f4cf851c7c51dd82634c534256412e" exitCode=0 Jul 15 13:07:02.655036 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.654934 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" event={"ID":"1e2749e6-96d8-4e06-8bdc-0693774149e9","Type":"ContainerDied","Data":"940e3efa188ca82552b131c77f67ff1063f4cf851c7c51dd82634c534256412e"} Jul 15 13:07:02.659873 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.659808 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" event={"ID":"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef","Type":"ContainerStarted","Data":"03c1822f96da6e0d604165fad9cf5926aa5a997861881fcda04ca0511f92cb00"} Jul 15 13:07:02.662285 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.661500 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghmn2" event={"ID":"429fe006-ddc8-454f-b468-9a6d3f553856","Type":"ContainerStarted","Data":"4a4bb95636126634105678822021350872a0d0f79695c31d75a282cac96daa3e"} Jul 15 13:07:02.666699 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.666675 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" event={"ID":"dac0d3ea-32ad-4873-a99d-340a8e421fb6","Type":"ContainerStarted","Data":"ae7640d0bee568072c87115b07161944390dac53ba436da783ea8ea4d0362769"} Jul 15 13:07:02.672024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.671971 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" event={"ID":"b709f9e1-85a0-4d82-adc8-17ee27958e0d","Type":"ContainerStarted","Data":"4ca6a8f08de386291e203ae78e71f3580067645e65afda204eded490e75e4ace"} Jul 15 13:07:02.990728 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:02.990625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:03.106318 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:03.106210 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:03.421316 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:03.420594 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:03.421316 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:03.420829 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:07:03.421316 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:03.420845 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:07:03.421316 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:03.420902 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:05.420881951 +0000 UTC m=+40.590828284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:07:03.521538 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:03.521501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:03.521745 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:03.521728 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:03.521870 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:03.521794 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:05.521776449 +0000 UTC m=+40.691722784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:07:03.688574 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:03.687795 2582 generic.go:356] "Generic (PLEG): container finished" podID="1e2749e6-96d8-4e06-8bdc-0693774149e9" containerID="d9fa9377dd23bde79e52e096118952f633fdb03001c81c4bfc0ca4ada817f4ff" exitCode=0 Jul 15 13:07:03.688574 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:03.687878 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" event={"ID":"1e2749e6-96d8-4e06-8bdc-0693774149e9","Type":"ContainerDied","Data":"d9fa9377dd23bde79e52e096118952f633fdb03001c81c4bfc0ca4ada817f4ff"} Jul 15 13:07:03.773160 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:03.773091 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:03.876801 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:03.875314 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:04.694970 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:04.694935 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" event={"ID":"1e2749e6-96d8-4e06-8bdc-0693774149e9","Type":"ContainerStarted","Data":"fd53f6ed75718b37d4ed1b5b6029acc17e8ba97f56a2f5cc73c8d681a925c9f9"} Jul 15 13:07:04.718849 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:04.718792 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jfl8z" podStartSLOduration=8.359695758 podStartE2EDuration="38.718773318s" totalImagesPullingTime="30.35907756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:04.718075325 +0000 UTC m=+39.888021682" watchObservedRunningTime="2026-07-15 13:07:04.718773318 +0000 UTC m=+39.888719674" Jul 15 13:07:04.842602 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:04.842549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:04.910161 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:04.910121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:05.447210 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:05.447175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:05.447396 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:05.447350 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:07:05.447396 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:05.447373 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:07:05.447505 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:05.447437 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:09.447416449 +0000 UTC m=+44.617362796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:07:05.548009 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:05.547966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:05.548230 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:05.548213 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:05.548320 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:05.548287 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:09.548266393 +0000 UTC m=+44.718212746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:07:08.809202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:08.809167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:08.902723 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:08.902679 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:09.476569 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:09.476538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:09.476766 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:09.476726 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:07:09.476766 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:09.476749 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:07:09.476869 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:09.476816 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:17.476794642 +0000 UTC m=+52.646740999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:07:09.576967 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:09.576935 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:09.577130 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:09.577094 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:09.577196 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:09.577170 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:17.577150696 +0000 UTC m=+52.747097045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:07:09.900420 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:09.900390 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:09.965182 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:09.965140 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:10.712434 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.712398 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" event={"ID":"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef","Type":"ContainerStarted","Data":"5bb9bfb803dae1de68bd3d6d44911efbf91166527995433ae9d1e018aec823a5"} Jul 15 13:07:10.712573 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.712444 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" event={"ID":"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef","Type":"ContainerStarted","Data":"7966d7156be101f93f28c86e56d92b698c1921a385409ef53319198240d2d541"} Jul 15 13:07:10.712573 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.712460 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" event={"ID":"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef","Type":"ContainerStarted","Data":"b127d0be52c83e1d50b69127123fee0ce41b92adf41794c020ef84d07d31e58c"} Jul 15 13:07:10.713906 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.713882 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghmn2" event={"ID":"429fe006-ddc8-454f-b468-9a6d3f553856","Type":"ContainerStarted","Data":"fabb85ddea623a61f6f2f69864a5eb89a8811819254569ab0936c2c43c4f3d16"} Jul 15 13:07:10.715343 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.715314 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" event={"ID":"dac0d3ea-32ad-4873-a99d-340a8e421fb6","Type":"ContainerStarted","Data":"c66806a3476af62a7d9493b4699cc8c78a88bb48bf47ef1695b6244e769a519d"} Jul 15 13:07:10.715457 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.715444 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:10.716866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.716845 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" event={"ID":"b709f9e1-85a0-4d82-adc8-17ee27958e0d","Type":"ContainerStarted","Data":"4881eee2b254a789632520790f21b65dc773ab3a964acaa6b253f8d8dfbbdd40"} Jul 15 13:07:10.717160 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.717144 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:07:10.733982 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.733944 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" podStartSLOduration=26.738500338 podStartE2EDuration="34.73393106s" totalImagesPullingTime="7.995430722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:10.733239237 +0000 UTC m=+45.903185608" watchObservedRunningTime="2026-07-15 13:07:10.73393106 +0000 UTC m=+45.903877391" Jul 15 13:07:10.748604 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.748558 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ghmn2" podStartSLOduration=1.799080189 podStartE2EDuration="9.748546724s" totalImagesPullingTime="7.949466535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:10.748293294 +0000 UTC m=+45.918239647" watchObservedRunningTime="2026-07-15 13:07:10.748546724 +0000 UTC m=+45.918493076" Jul 15 13:07:10.767454 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.767417 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" podStartSLOduration=26.780008012 podStartE2EDuration="34.767385123s" totalImagesPullingTime="7.987377111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:10.766954284 +0000 UTC m=+45.936900638" watchObservedRunningTime="2026-07-15 13:07:10.767385123 +0000 UTC m=+45.937331468" Jul 15 13:07:10.784313 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:10.784271 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" podStartSLOduration=26.789509444 podStartE2EDuration="34.7842385s" totalImagesPullingTime="7.994729056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:10.783196508 +0000 UTC m=+45.953142849" watchObservedRunningTime="2026-07-15 13:07:10.7842385 +0000 UTC m=+45.954184854" Jul 15 13:07:12.622755 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:12.622707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:12.723751 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:12.723701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:15.920758 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:15.920708 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:15.983835 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:15.983810 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:17.531032 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:17.530996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:17.531484 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:17.531118 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:07:17.531484 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:17.531134 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:07:17.531484 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:17.531205 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:33.531178568 +0000 UTC m=+68.701124899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:07:17.631507 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:17.631477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:17.631625 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:17.631610 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:17.631674 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:17.631658 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:33.631645885 +0000 UTC m=+68.801592216 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:07:19.072943 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:19.072907 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:21.828971 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:21.828940 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:25.492118 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:25.492086 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:27.647394 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:27.647362 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwvqz" Jul 15 13:07:27.801439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:27.801402 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:27.902053 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:27.901976 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:31.024373 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.024339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:07:31.026720 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.026696 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 13:07:31.038771 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.038751 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62b177c1-f250-4a0e-9cbb-609fc3ffc741-original-pull-secret\") pod \"global-pull-secret-syncer-6hjrv\" (UID: \"62b177c1-f250-4a0e-9cbb-609fc3ffc741\") " pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:07:31.125576 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.125551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:07:31.127653 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.127637 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 13:07:31.136224 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:31.136207 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 13:07:31.136296 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:31.136253 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:08:35.136238434 +0000 UTC m=+130.306184778 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : secret "metrics-daemon-secret" not found Jul 15 13:07:31.226573 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.226553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:07:31.229219 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.229203 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 13:07:31.239383 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.239365 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 13:07:31.249128 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.249107 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj62v\" (UniqueName: \"kubernetes.io/projected/7154e2c8-357d-41e0-83c4-f2cc07b530ad-kube-api-access-qj62v\") pod \"network-check-target-89wts\" (UID: \"7154e2c8-357d-41e0-83c4-f2cc07b530ad\") " pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:07:31.274286 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.274267 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nr6v2\"" Jul 15 13:07:31.277153 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.277120 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hjrv" Jul 15 13:07:31.282801 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.282785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:07:31.404059 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.404026 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89wts"] Jul 15 13:07:31.407214 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:07:31.407185 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7154e2c8_357d_41e0_83c4_f2cc07b530ad.slice/crio-4256dff335ee8f2dbd9dabc46f8ff05cddaf143ab1360279f6670cbf2d4212de WatchSource:0}: Error finding container 4256dff335ee8f2dbd9dabc46f8ff05cddaf143ab1360279f6670cbf2d4212de: Status 404 returned error can't find the container with id 4256dff335ee8f2dbd9dabc46f8ff05cddaf143ab1360279f6670cbf2d4212de Jul 15 13:07:31.420814 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.420796 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6hjrv"] Jul 15 13:07:31.422879 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:07:31.422857 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b177c1_f250_4a0e_9cbb_609fc3ffc741.slice/crio-c0408b9a844e0809016b0a6a25e0dd7e211145a8da15f9343e4251bcbd6af9f3 WatchSource:0}: Error finding container c0408b9a844e0809016b0a6a25e0dd7e211145a8da15f9343e4251bcbd6af9f3: Status 404 returned error can't find the container with id c0408b9a844e0809016b0a6a25e0dd7e211145a8da15f9343e4251bcbd6af9f3 Jul 15 13:07:31.778278 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.778248 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6hjrv" event={"ID":"62b177c1-f250-4a0e-9cbb-609fc3ffc741","Type":"ContainerStarted","Data":"c0408b9a844e0809016b0a6a25e0dd7e211145a8da15f9343e4251bcbd6af9f3"} Jul 15 13:07:31.779317 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:31.779287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89wts" event={"ID":"7154e2c8-357d-41e0-83c4-f2cc07b530ad","Type":"ContainerStarted","Data":"4256dff335ee8f2dbd9dabc46f8ff05cddaf143ab1360279f6670cbf2d4212de"} Jul 15 13:07:33.543926 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:33.543886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:07:33.544388 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:33.544039 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:07:33.544388 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:33.544058 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:07:33.544388 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:33.544118 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:08:05.544098919 +0000 UTC m=+100.714045250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:07:33.645295 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:33.645263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:07:33.645451 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:33.645429 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:33.645568 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:07:33.645507 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:08:05.645483051 +0000 UTC m=+100.815429398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:07:34.901264 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:34.901231 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:34.998836 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:34.998799 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:36.038180 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:36.038144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:36.115469 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:36.115436 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:36.801196 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:36.801160 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6hjrv" event={"ID":"62b177c1-f250-4a0e-9cbb-609fc3ffc741","Type":"ContainerStarted","Data":"5e81d1a8c276fdbddc28687b239bb695f7edbe4bf77921d72a5d79043e9a66d4"} Jul 15 13:07:36.802564 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:36.802541 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89wts" event={"ID":"7154e2c8-357d-41e0-83c4-f2cc07b530ad","Type":"ContainerStarted","Data":"3006ea8f0ab4bc690907fb41e8e02f9534b996a9b8bfac14713a3858fba94ad0"} Jul 15 13:07:36.802685 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:36.802585 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:07:36.816533 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:36.816493 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6hjrv" podStartSLOduration=66.110741964 podStartE2EDuration="1m10.816476716s" totalImagesPullingTime="4.705734752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:36.816161789 +0000 UTC m=+71.986108142" watchObservedRunningTime="2026-07-15 13:07:36.816476716 +0000 UTC m=+71.986423071" Jul 15 13:07:41.897851 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:41.897810 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:07:41.965863 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:07:41.965834 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:08:03.860844 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:08:03.860733 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:08:03.967118 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:08:03.967089 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:08:05.571251 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:08:05.571219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:08:05.571643 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:08:05.571390 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:08:05.571643 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:08:05.571408 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:08:05.571643 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:08:05.571477 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:09:09.571459608 +0000 UTC m=+164.741405944 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:08:05.672472 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:08:05.672437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:08:05.672592 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:08:05.672549 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:08:05.672629 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:08:05.672604 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:09:09.672591563 +0000 UTC m=+164.842537894 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:08:07.808623 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:08:07.808593 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-89wts" Jul 15 13:08:07.828124 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:08:07.828078 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-89wts" podStartSLOduration=97.111626491 podStartE2EDuration="1m41.828064325s" totalImagesPullingTime="4.716437834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:36.830799941 +0000 UTC m=+72.000746326" watchObservedRunningTime="2026-07-15 13:08:07.828064325 +0000 UTC m=+102.998010679" Jul 15 13:08:35.176103 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:08:35.176064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:08:35.176579 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:08:35.176187 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 13:08:35.176579 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:08:35.176240 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:37.176226374 +0000 UTC m=+252.346172705 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : secret "metrics-daemon-secret" not found Jul 15 13:08:45.348298 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:08:45.348267 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:09:04.757817 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:09:04.757763 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" podUID="e0657627-2d08-41e9-9326-0f7fc596f470" Jul 15 13:09:04.766896 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:09:04.766867 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mjlh5" podUID="e2e9cb69-d4ea-4457-9b22-29286c5cc772" Jul 15 13:09:05.046164 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:05.046079 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:09:05.046324 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:05.046079 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mjlh5" Jul 15 13:09:05.466558 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:09:05.466526 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:09:09.617090 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:09.617053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:09:09.617533 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:09:09.617172 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:09:09.617533 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:09:09.617183 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77d7b747c9-fcj99: secret "image-registry-tls" not found Jul 15 13:09:09.617533 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:09:09.617238 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls podName:e0657627-2d08-41e9-9326-0f7fc596f470 nodeName:}" failed. No retries permitted until 2026-07-15 13:11:11.617224207 +0000 UTC m=+286.787170538 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls") pod "image-registry-77d7b747c9-fcj99" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470") : secret "image-registry-tls" not found Jul 15 13:09:09.717319 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:09.717291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:09:09.717440 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:09:09.717408 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:09:09.717482 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:09:09.717450 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls podName:e2e9cb69-d4ea-4457-9b22-29286c5cc772 nodeName:}" failed. No retries permitted until 2026-07-15 13:11:11.717437724 +0000 UTC m=+286.887384056 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls") pod "dns-default-mjlh5" (UID: "e2e9cb69-d4ea-4457-9b22-29286c5cc772") : secret "dns-default-metrics-tls" not found Jul 15 13:09:10.716003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:10.715971 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" podUID="dac0d3ea-32ad-4873-a99d-340a8e421fb6" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.6:8000/readyz\": dial tcp 10.132.0.6:8000: connect: connection refused" Jul 15 13:09:11.063424 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:11.063352 2582 generic.go:356] "Generic (PLEG): container finished" podID="dac0d3ea-32ad-4873-a99d-340a8e421fb6" containerID="c66806a3476af62a7d9493b4699cc8c78a88bb48bf47ef1695b6244e769a519d" exitCode=1 Jul 15 13:09:11.063573 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:11.063434 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" event={"ID":"dac0d3ea-32ad-4873-a99d-340a8e421fb6","Type":"ContainerDied","Data":"c66806a3476af62a7d9493b4699cc8c78a88bb48bf47ef1695b6244e769a519d"} Jul 15 13:09:11.063793 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:11.063772 2582 scope.go:117] "RemoveContainer" containerID="c66806a3476af62a7d9493b4699cc8c78a88bb48bf47ef1695b6244e769a519d" Jul 15 13:09:11.065013 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:11.064898 2582 generic.go:356] "Generic (PLEG): container finished" podID="b709f9e1-85a0-4d82-adc8-17ee27958e0d" containerID="4881eee2b254a789632520790f21b65dc773ab3a964acaa6b253f8d8dfbbdd40" exitCode=255 Jul 15 13:09:11.065013 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:11.064940 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" event={"ID":"b709f9e1-85a0-4d82-adc8-17ee27958e0d","Type":"ContainerDied","Data":"4881eee2b254a789632520790f21b65dc773ab3a964acaa6b253f8d8dfbbdd40"} Jul 15 13:09:11.065188 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:11.065173 2582 scope.go:117] "RemoveContainer" containerID="4881eee2b254a789632520790f21b65dc773ab3a964acaa6b253f8d8dfbbdd40" Jul 15 13:09:12.012223 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:12.012190 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:09:12.029276 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:12.029251 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:09:12.069354 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:12.069328 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" event={"ID":"dac0d3ea-32ad-4873-a99d-340a8e421fb6","Type":"ContainerStarted","Data":"a4498ab315ba6669ea086e38e9974ee10637485f32ecdd3da40e7913405dc682"} Jul 15 13:09:12.069564 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:12.069537 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:09:12.070034 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:12.070018 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:09:12.071109 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:12.071088 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" event={"ID":"b709f9e1-85a0-4d82-adc8-17ee27958e0d","Type":"ContainerStarted","Data":"a2741c2f095a2b5f3414546aba5df231164aa4a1b5987460fce9f86e66f674f2"} Jul 15 13:09:19.455594 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:19.455558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:09:41.784068 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:09:41.784035 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:27.591655 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:27.591618 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:30.653969 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:30.653937 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:30.746707 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:30.746680 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:30.850650 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:30.850624 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:31.819254 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.819218 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vxkpv"] Jul 15 13:10:31.821357 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.821340 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.823476 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.823447 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 13:10:31.824045 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.824028 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 13:10:31.824469 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.824449 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 13:10:31.824995 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.824976 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m9psk\"" Jul 15 13:10:31.825411 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.825394 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 13:10:31.836958 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.836939 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vxkpv"] Jul 15 13:10:31.863739 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.863699 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkc6p\" (UniqueName: \"kubernetes.io/projected/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-api-access-pkc6p\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.863838 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.863748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6e424f9-072b-487b-b86d-50a57cd8bbc6-crio-socket\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.863838 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.863769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.863923 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.863845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.863923 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.863906 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6e424f9-072b-487b-b86d-50a57cd8bbc6-data-volume\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.866679 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.866662 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:31.937173 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.937145 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:31.964341 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.964318 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6e424f9-072b-487b-b86d-50a57cd8bbc6-crio-socket\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.964426 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.964352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.964426 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.964414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.964533 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.964424 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6e424f9-072b-487b-b86d-50a57cd8bbc6-crio-socket\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.964533 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:31.964504 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:31.964533 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:31.964522 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:31.964685 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:31.964554 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:32.464538724 +0000 UTC m=+247.634485055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:31.964685 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:31.964590 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:32.46457278 +0000 UTC m=+247.634519113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : configmap "kube-rbac-proxy" not found Jul 15 13:10:31.964685 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.964609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6e424f9-072b-487b-b86d-50a57cd8bbc6-data-volume\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.964685 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.964645 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkc6p\" (UniqueName: \"kubernetes.io/projected/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-api-access-pkc6p\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.964907 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.964890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6e424f9-072b-487b-b86d-50a57cd8bbc6-data-volume\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:31.972990 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:31.972970 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkc6p\" (UniqueName: \"kubernetes.io/projected/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-api-access-pkc6p\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:32.467629 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:32.467604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:32.467796 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:32.467651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:32.467796 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:32.467705 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:32.467796 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:32.467748 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:32.467796 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:32.467787 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:33.467770553 +0000 UTC m=+248.637716900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : configmap "kube-rbac-proxy" not found Jul 15 13:10:32.467937 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:32.467802 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:33.467794893 +0000 UTC m=+248.637741224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:32.752331 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:32.752233 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:32.848775 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:32.848736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:33.185910 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:33.185830 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:33.253176 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:33.253145 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:33.405860 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:33.405824 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:33.476577 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:33.476549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:33.476753 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:33.476599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:33.476753 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:33.476661 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:33.476753 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:33.476696 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:33.476753 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:33.476702 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:35.47668919 +0000 UTC m=+250.646635520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : configmap "kube-rbac-proxy" not found Jul 15 13:10:33.476909 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:33.476760 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:35.476748642 +0000 UTC m=+250.646694973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:33.499793 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:33.499767 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:34.812965 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:34.812937 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjvdt_57661864-e119-4734-9e4b-f9932acf74e4/dns-node-resolver/0.log" Jul 15 13:10:35.090839 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:35.090765 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:35.176883 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:35.176854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:35.492262 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:35.492226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:35.492468 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:35.492289 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:35.492468 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:35.492373 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:35.492468 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:35.492442 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:39.492416468 +0000 UTC m=+254.662362799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : configmap "kube-rbac-proxy" not found Jul 15 13:10:35.492468 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:35.492374 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:35.492684 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:35.492513 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:39.49249859 +0000 UTC m=+254.662444922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:35.614081 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:35.614052 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gptn8_4736cf14-4df8-4a5d-95c1-d026e0d88d42/node-ca/0.log" Jul 15 13:10:36.212894 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:36.212867 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghmn2_429fe006-ddc8-454f-b468-9a6d3f553856/serve-healthcheck-canary/0.log" Jul 15 13:10:36.296818 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:36.296788 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:37.203896 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:37.203853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:10:37.204070 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:37.203996 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 13:10:37.204070 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:37.204061 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs podName:b811d4da-54d9-42e2-b09c-ada3620a5a92 nodeName:}" failed. No retries permitted until 2026-07-15 13:12:39.204045363 +0000 UTC m=+374.373991698 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs") pod "network-metrics-daemon-cx6mz" (UID: "b811d4da-54d9-42e2-b09c-ada3620a5a92") : secret "metrics-daemon-secret" not found Jul 15 13:10:38.140314 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:38.140285 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:38.438529 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:38.438461 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:38.523350 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:38.523320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:38.630389 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:38.630365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:38.835272 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:38.835249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:39.521885 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:39.521858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:39.522216 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:39.521907 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:39.522216 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:39.521992 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:39.522216 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:39.522000 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:39.522216 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:39.522038 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:47.52202568 +0000 UTC m=+262.691972011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : configmap "kube-rbac-proxy" not found Jul 15 13:10:39.522216 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:39.522050 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:47.522044546 +0000 UTC m=+262.691990877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:39.844559 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:39.844500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:40.257475 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:40.257448 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:40.469928 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:40.469904 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:40.638302 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:40.638239 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:40.838689 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:40.838661 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:44.718690 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:44.718659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:44.818856 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:44.818831 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:44.919168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:44.919144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:45.635879 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:45.635846 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:46.038930 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:46.038902 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:46.245831 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:46.245807 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:46.634889 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:46.634863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:47.273759 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:47.273681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:47.578942 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:47.578878 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:47.579070 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:47.578940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:47.579070 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:47.579036 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:47.579158 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:10:47.579092 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls podName:d6e424f9-072b-487b-b86d-50a57cd8bbc6 nodeName:}" failed. No retries permitted until 2026-07-15 13:11:03.579079234 +0000 UTC m=+278.749025585 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vxkpv" (UID: "d6e424f9-072b-487b-b86d-50a57cd8bbc6") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:47.579344 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:47.579327 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6e424f9-072b-487b-b86d-50a57cd8bbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:10:47.830881 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:47.830812 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:47.981983 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:47.981956 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:48.052664 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:48.052643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:48.516504 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:48.516476 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:48.636603 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:48.636578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:48.979477 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:48.979451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:49.286950 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:49.286887 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:51.152160 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:51.152130 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:51.253439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:51.253410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:51.352160 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:51.352132 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:51.657634 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:51.657611 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:51.826510 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:51.826482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:52.470598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:52.470568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:52.566749 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:52.566709 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:52.836600 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:52.836538 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:53.036458 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:53.036427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:53.237636 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:53.237608 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:53.339844 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:53.339817 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:56.315136 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:56.315103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:56.407397 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:56.407369 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:57.671639 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:57.671606 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:10:57.774111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:10:57.774085 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:01.617666 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:01.617633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:01.717958 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:01.717934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:01.804166 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:01.804142 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:02.141173 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:02.141148 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:02.257819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:02.257789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:02.321788 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:02.321763 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:03.587144 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:03.587107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:11:03.589279 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:03.589256 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6e424f9-072b-487b-b86d-50a57cd8bbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxkpv\" (UID: \"d6e424f9-072b-487b-b86d-50a57cd8bbc6\") " pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:11:03.630251 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:03.630221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vxkpv" Jul 15 13:11:03.748165 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:03.748138 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vxkpv"] Jul 15 13:11:03.751466 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:11:03.751431 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e424f9_072b_487b_b86d_50a57cd8bbc6.slice/crio-2cd894dc557ee69fe0e91f8ae44a9134683de3299bbb52bddb0d3a17e6374dba WatchSource:0}: Error finding container 2cd894dc557ee69fe0e91f8ae44a9134683de3299bbb52bddb0d3a17e6374dba: Status 404 returned error can't find the container with id 2cd894dc557ee69fe0e91f8ae44a9134683de3299bbb52bddb0d3a17e6374dba Jul 15 13:11:04.035725 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:04.035694 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:04.106236 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:04.106209 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:04.389104 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:04.389019 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vxkpv" event={"ID":"d6e424f9-072b-487b-b86d-50a57cd8bbc6","Type":"ContainerStarted","Data":"2cd894dc557ee69fe0e91f8ae44a9134683de3299bbb52bddb0d3a17e6374dba"} Jul 15 13:11:04.554342 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:04.554293 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:04.626107 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:04.626073 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:04.816085 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:04.816043 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:04.887314 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:04.887282 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:05.633932 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:05.633897 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:07.660667 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:07.660633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:07.760810 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:07.760777 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:08.046765 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:08.046709 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" podUID="e0657627-2d08-41e9-9326-0f7fc596f470" Jul 15 13:11:08.046765 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:08.046726 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mjlh5" podUID="e2e9cb69-d4ea-4457-9b22-29286c5cc772" Jul 15 13:11:08.401931 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:08.401854 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:11:08.401931 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:08.401882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mjlh5" Jul 15 13:11:09.407580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:09.407533 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vxkpv" event={"ID":"d6e424f9-072b-487b-b86d-50a57cd8bbc6","Type":"ContainerStarted","Data":"0d74b81b276ed0e1dc62eb8913c56a7c48e18a31ff6ff5ef8d2a2f71a527cbd4"} Jul 15 13:11:10.252248 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:10.252202 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:10.323905 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:10.323874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:11.416678 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.416597 2582 generic.go:356] "Generic (PLEG): container finished" podID="dac0d3ea-32ad-4873-a99d-340a8e421fb6" containerID="a4498ab315ba6669ea086e38e9974ee10637485f32ecdd3da40e7913405dc682" exitCode=1 Jul 15 13:11:11.416678 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.416654 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" event={"ID":"dac0d3ea-32ad-4873-a99d-340a8e421fb6","Type":"ContainerDied","Data":"a4498ab315ba6669ea086e38e9974ee10637485f32ecdd3da40e7913405dc682"} Jul 15 13:11:11.417168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.416732 2582 scope.go:117] "RemoveContainer" containerID="c66806a3476af62a7d9493b4699cc8c78a88bb48bf47ef1695b6244e769a519d" Jul 15 13:11:11.417168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.417036 2582 scope.go:117] "RemoveContainer" containerID="a4498ab315ba6669ea086e38e9974ee10637485f32ecdd3da40e7913405dc682" Jul 15 13:11:11.417284 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:11.417260 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b_open-cluster-management-agent-addon(dac0d3ea-32ad-4873-a99d-340a8e421fb6)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" podUID="dac0d3ea-32ad-4873-a99d-340a8e421fb6" Jul 15 13:11:11.653554 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.653511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:11:11.656090 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.656060 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"image-registry-77d7b747c9-fcj99\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:11:11.705519 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.705493 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j4np7\"" Jul 15 13:11:11.714157 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.714138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:11:11.754340 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.754314 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:11:11.756860 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:11.756838 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2e9cb69-d4ea-4457-9b22-29286c5cc772-metrics-tls\") pod \"dns-default-mjlh5\" (UID: \"e2e9cb69-d4ea-4457-9b22-29286c5cc772\") " pod="openshift-dns/dns-default-mjlh5" Jul 15 13:11:12.004581 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.004558 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-txt24\"" Jul 15 13:11:12.012179 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.012160 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:11:12.012462 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.012443 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77d7b747c9-fcj99"] Jul 15 13:11:12.013246 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.013230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mjlh5" Jul 15 13:11:12.015007 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:11:12.014991 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0657627_2d08_41e9_9326_0f7fc596f470.slice/crio-58433cd32bf4f453215e27aa72d56887d5a3050e9e4754543d1bdf4df71df98a WatchSource:0}: Error finding container 58433cd32bf4f453215e27aa72d56887d5a3050e9e4754543d1bdf4df71df98a: Status 404 returned error can't find the container with id 58433cd32bf4f453215e27aa72d56887d5a3050e9e4754543d1bdf4df71df98a Jul 15 13:11:12.030192 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.030166 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" podUID="b709f9e1-85a0-4d82-adc8-17ee27958e0d" containerName="addon-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/healthz\": dial tcp 10.132.0.7:8000: connect: connection refused" Jul 15 13:11:12.069693 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.069665 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:11:12.138579 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.138549 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mjlh5"] Jul 15 13:11:12.140841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:11:12.140817 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e9cb69_d4ea_4457_9b22_29286c5cc772.slice/crio-d2bb7e6950f5572c10cea51d701368852692473b79f0cc96fdb26bf488915099 WatchSource:0}: Error finding container d2bb7e6950f5572c10cea51d701368852692473b79f0cc96fdb26bf488915099: Status 404 returned error can't find the container with id d2bb7e6950f5572c10cea51d701368852692473b79f0cc96fdb26bf488915099 Jul 15 13:11:12.422141 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.422066 2582 scope.go:117] "RemoveContainer" containerID="a4498ab315ba6669ea086e38e9974ee10637485f32ecdd3da40e7913405dc682" Jul 15 13:11:12.422533 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:12.422283 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b_open-cluster-management-agent-addon(dac0d3ea-32ad-4873-a99d-340a8e421fb6)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" podUID="dac0d3ea-32ad-4873-a99d-340a8e421fb6" Jul 15 13:11:12.423653 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.423633 2582 generic.go:356] "Generic (PLEG): container finished" podID="b709f9e1-85a0-4d82-adc8-17ee27958e0d" containerID="a2741c2f095a2b5f3414546aba5df231164aa4a1b5987460fce9f86e66f674f2" exitCode=255 Jul 15 13:11:12.423765 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.423729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" event={"ID":"b709f9e1-85a0-4d82-adc8-17ee27958e0d","Type":"ContainerDied","Data":"a2741c2f095a2b5f3414546aba5df231164aa4a1b5987460fce9f86e66f674f2"} Jul 15 13:11:12.423831 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.423786 2582 scope.go:117] "RemoveContainer" containerID="4881eee2b254a789632520790f21b65dc773ab3a964acaa6b253f8d8dfbbdd40" Jul 15 13:11:12.424099 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.424080 2582 scope.go:117] "RemoveContainer" containerID="a2741c2f095a2b5f3414546aba5df231164aa4a1b5987460fce9f86e66f674f2" Jul 15 13:11:12.424251 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:12.424224 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-7cb7bd6b76-sf956_open-cluster-management-agent-addon(b709f9e1-85a0-4d82-adc8-17ee27958e0d)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" podUID="b709f9e1-85a0-4d82-adc8-17ee27958e0d" Jul 15 13:11:12.425425 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.425402 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" event={"ID":"e0657627-2d08-41e9-9326-0f7fc596f470","Type":"ContainerStarted","Data":"5fda98f02122fca83c581bc266cf7b8f796f76b4529bf7b10e8a0ffd4b4c063f"} Jul 15 13:11:12.425510 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.425448 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:11:12.425510 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.425467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" event={"ID":"e0657627-2d08-41e9-9326-0f7fc596f470","Type":"ContainerStarted","Data":"58433cd32bf4f453215e27aa72d56887d5a3050e9e4754543d1bdf4df71df98a"} Jul 15 13:11:12.427290 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.427263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vxkpv" event={"ID":"d6e424f9-072b-487b-b86d-50a57cd8bbc6","Type":"ContainerStarted","Data":"c0e15d61fed084b7bf0b69ee90cc861ec66ff2dac6ef88ff8bcb4c7b93b5e524"} Jul 15 13:11:12.428407 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.428390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjlh5" event={"ID":"e2e9cb69-d4ea-4457-9b22-29286c5cc772","Type":"ContainerStarted","Data":"d2bb7e6950f5572c10cea51d701368852692473b79f0cc96fdb26bf488915099"} Jul 15 13:11:12.478352 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.478296 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" podStartSLOduration=286.478278193 podStartE2EDuration="4m46.478278193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:12.477029217 +0000 UTC m=+287.646975572" watchObservedRunningTime="2026-07-15 13:11:12.478278193 +0000 UTC m=+287.648224547" Jul 15 13:11:12.492521 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.492475 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:12.592537 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:12.592514 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:13.433795 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:13.433764 2582 scope.go:117] "RemoveContainer" containerID="a4498ab315ba6669ea086e38e9974ee10637485f32ecdd3da40e7913405dc682" Jul 15 13:11:13.434226 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:13.433955 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b_open-cluster-management-agent-addon(dac0d3ea-32ad-4873-a99d-340a8e421fb6)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" podUID="dac0d3ea-32ad-4873-a99d-340a8e421fb6" Jul 15 13:11:16.446978 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:16.446948 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vxkpv" event={"ID":"d6e424f9-072b-487b-b86d-50a57cd8bbc6","Type":"ContainerStarted","Data":"6396d666c21c610ddceeb32d91c0ed82d3c7111391418e361c3e5fb200e092dd"} Jul 15 13:11:16.448700 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:16.448676 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjlh5" event={"ID":"e2e9cb69-d4ea-4457-9b22-29286c5cc772","Type":"ContainerStarted","Data":"9bb9ca5924e75093fd4c2fd599b6d333367e35c7564b937d1c90d0c54b563c6d"} Jul 15 13:11:16.448800 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:16.448740 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mjlh5" event={"ID":"e2e9cb69-d4ea-4457-9b22-29286c5cc772","Type":"ContainerStarted","Data":"6023ff14e997c40cb4299c19ce18816c6c806759a518f86310224903e1618d9d"} Jul 15 13:11:16.448800 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:16.448762 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mjlh5" Jul 15 13:11:16.464681 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:16.464586 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vxkpv" podStartSLOduration=33.905668395 podStartE2EDuration="45.464571079s" totalImagesPullingTime="11.558902684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:10:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:16.463863566 +0000 UTC m=+291.633809919" watchObservedRunningTime="2026-07-15 13:11:16.464571079 +0000 UTC m=+291.634517433" Jul 15 13:11:16.479553 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:16.479518 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mjlh5" podStartSLOduration=251.346456635 podStartE2EDuration="4m15.479506199s" totalImagesPullingTime="4.133049564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:16.479222086 +0000 UTC m=+291.649168439" watchObservedRunningTime="2026-07-15 13:11:16.479506199 +0000 UTC m=+291.649452551" Jul 15 13:11:17.365539 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:17.365503 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:20.148017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:20.147979 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:22.029401 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:22.029371 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" Jul 15 13:11:22.029878 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:22.029759 2582 scope.go:117] "RemoveContainer" containerID="a2741c2f095a2b5f3414546aba5df231164aa4a1b5987460fce9f86e66f674f2" Jul 15 13:11:22.456285 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:22.456257 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cx6mz" podUID="b811d4da-54d9-42e2-b09c-ada3620a5a92" Jul 15 13:11:22.468497 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:22.468464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cb7bd6b76-sf956" event={"ID":"b709f9e1-85a0-4d82-adc8-17ee27958e0d","Type":"ContainerStarted","Data":"9da58d73d6a3437012e7d6af2d4baa2ec4de1250d5b0cca31adb8cdbe65083cf"} Jul 15 13:11:24.531471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:24.531439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:25.224785 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:25.224754 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:11:25.224785 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:25.224779 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:11:25.228337 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:25.228319 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 13:11:25.949406 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:25.949375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:26.172383 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:26.172353 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:26.453468 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:26.453436 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mjlh5" Jul 15 13:11:26.851260 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:26.851180 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:26.966889 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:26.966856 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:27.455505 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:27.455473 2582 scope.go:117] "RemoveContainer" containerID="a4498ab315ba6669ea086e38e9974ee10637485f32ecdd3da40e7913405dc682" Jul 15 13:11:28.417097 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:28.417063 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77d7b747c9-fcj99"] Jul 15 13:11:28.421510 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:28.421467 2582 patch_prober.go:28] interesting pod/image-registry-77d7b747c9-fcj99 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 15 13:11:28.421647 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:28.421544 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" podUID="e0657627-2d08-41e9-9326-0f7fc596f470" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:11:28.487792 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:28.487760 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" event={"ID":"dac0d3ea-32ad-4873-a99d-340a8e421fb6","Type":"ContainerStarted","Data":"2a40f235446daec0cdefac21fd532700a1055119080cc3418875eb1ddec7b7f9"} Jul 15 13:11:28.488063 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:28.488043 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:11:28.489504 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:28.489486 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5fc5c5c9-7cw2b" Jul 15 13:11:28.553204 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:28.553173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:29.873428 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:29.873398 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:31.132249 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.132222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:31.227935 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.227906 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:31.827685 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.827653 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:31.881588 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.881558 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xskc4"] Jul 15 13:11:31.886825 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.886809 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.889279 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.889259 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 13:11:31.889279 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.889263 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 13:11:31.889471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.889302 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 13:11:31.889747 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.889730 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fx726\"" Jul 15 13:11:31.890328 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.890312 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 13:11:31.891251 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.891232 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 13:11:31.892129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.892114 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 13:11:31.947284 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.947254 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:31.996210 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.996190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-root\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.996358 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.996341 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.996473 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.996460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-wtmp\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.996563 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.996552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb13fb50-fa90-4093-b62f-d7d89ab1db54-metrics-client-ca\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.996652 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.996640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-accelerators-collector-config\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.996773 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.996760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ncf\" (UniqueName: \"kubernetes.io/projected/bb13fb50-fa90-4093-b62f-d7d89ab1db54-kube-api-access-d6ncf\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.996898 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.996886 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-textfile\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.997083 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.997048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-sys\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:31.997083 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:31.997080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-tls\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.042891 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.042864 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:32.098385 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ncf\" (UniqueName: \"kubernetes.io/projected/bb13fb50-fa90-4093-b62f-d7d89ab1db54-kube-api-access-d6ncf\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098385 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-textfile\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-sys\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-tls\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098465 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-sys\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098548 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:32.098480 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 13:11:32.098548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-root\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098548 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:11:32.098542 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-tls podName:bb13fb50-fa90-4093-b62f-d7d89ab1db54 nodeName:}" failed. No retries permitted until 2026-07-15 13:11:32.598528652 +0000 UTC m=+307.768474984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-tls") pod "node-exporter-xskc4" (UID: "bb13fb50-fa90-4093-b62f-d7d89ab1db54") : secret "node-exporter-tls" not found Jul 15 13:11:32.098871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-root\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-wtmp\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb13fb50-fa90-4093-b62f-d7d89ab1db54-metrics-client-ca\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-accelerators-collector-config\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-textfile\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.098871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.098832 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-wtmp\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.099220 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.099203 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-accelerators-collector-config\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.099259 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.099208 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb13fb50-fa90-4093-b62f-d7d89ab1db54-metrics-client-ca\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.100753 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.100735 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.106277 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.106254 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ncf\" (UniqueName: \"kubernetes.io/projected/bb13fb50-fa90-4093-b62f-d7d89ab1db54-kube-api-access-d6ncf\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.143040 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.143016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:32.246519 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.246495 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:32.380266 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.380207 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:32.510919 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.510895 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:32.602065 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.602038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-tls\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.604189 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.604163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb13fb50-fa90-4093-b62f-d7d89ab1db54-node-exporter-tls\") pod \"node-exporter-xskc4\" (UID: \"bb13fb50-fa90-4093-b62f-d7d89ab1db54\") " pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.753797 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.753766 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:32.796467 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.796447 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xskc4" Jul 15 13:11:32.807278 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:11:32.807247 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb13fb50_fa90_4093_b62f_d7d89ab1db54.slice/crio-0f3f436a3c199facc437d749a22c81bde6c70e012adf6d9b31fce993384eead8 WatchSource:0}: Error finding container 0f3f436a3c199facc437d749a22c81bde6c70e012adf6d9b31fce993384eead8: Status 404 returned error can't find the container with id 0f3f436a3c199facc437d749a22c81bde6c70e012adf6d9b31fce993384eead8 Jul 15 13:11:32.809071 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.809055 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:11:32.855649 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.855624 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:32.958209 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:32.958185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:33.044677 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:33.044614 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:33.156397 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:33.156372 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:33.258383 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:33.258353 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:33.507049 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:33.507009 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xskc4" event={"ID":"bb13fb50-fa90-4093-b62f-d7d89ab1db54","Type":"ContainerStarted","Data":"0f3f436a3c199facc437d749a22c81bde6c70e012adf6d9b31fce993384eead8"} Jul 15 13:11:34.362922 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:34.362890 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:34.435555 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:34.435525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:34.455249 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:34.455223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:11:34.510872 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:34.510843 2582 generic.go:356] "Generic (PLEG): container finished" podID="bb13fb50-fa90-4093-b62f-d7d89ab1db54" containerID="f6581d70593995e2650ef3955ac5f994c877f7426fc7fe69b387438f70cfaa95" exitCode=0 Jul 15 13:11:34.510981 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:34.510886 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xskc4" event={"ID":"bb13fb50-fa90-4093-b62f-d7d89ab1db54","Type":"ContainerDied","Data":"f6581d70593995e2650ef3955ac5f994c877f7426fc7fe69b387438f70cfaa95"} Jul 15 13:11:34.518597 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:34.518572 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:34.881526 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:34.881446 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:34.983949 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:34.983879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:35.058237 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:35.058200 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:35.314804 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:35.314771 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:35.516532 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:35.516495 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xskc4" event={"ID":"bb13fb50-fa90-4093-b62f-d7d89ab1db54","Type":"ContainerStarted","Data":"a12363f8f4df13857b7c40e648595d75b912d58b86dee3a01574cb79cd7223da"} Jul 15 13:11:35.516945 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:35.516558 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xskc4" event={"ID":"bb13fb50-fa90-4093-b62f-d7d89ab1db54","Type":"ContainerStarted","Data":"ee1ab58f4212dc38b7f7f148d21b5012608851e11e39ec602e44d2c87fd09e9f"} Jul 15 13:11:35.539383 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:35.539334 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xskc4" podStartSLOduration=3.583129023 podStartE2EDuration="4.539317624s" totalImagesPullingTime="893.029469ms" totalInitContainerRuntime="63.159132ms" isStatefulPod=true podCreationTimestamp="2026-07-15 13:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:35.538144322 +0000 UTC m=+310.708090688" watchObservedRunningTime="2026-07-15 13:11:35.539317624 +0000 UTC m=+310.709263977" Jul 15 13:11:35.611768 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:35.611673 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:35.695860 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:35.695811 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:36.738769 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:36.738736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:36.814036 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:36.814007 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:36.961390 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:36.961367 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:37.063673 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:37.063621 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:37.524996 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:37.524969 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:37.623105 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:37.623075 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:38.125280 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:38.125242 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:38.422651 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:38.422592 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:11:42.039114 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:42.039079 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" podUID="a59d4b16-44a0-41ba-89ee-ac8ef6d991ef" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 13:11:44.632954 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:44.632925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:44.813198 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:44.813167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:48.620881 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:48.620852 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:48.683620 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:48.683594 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:48.777840 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:48.777811 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:48.855673 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:48.855641 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:52.038569 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:52.038530 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" podUID="a59d4b16-44a0-41ba-89ee-ac8ef6d991ef" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 13:11:52.859114 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:52.859079 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:53.018219 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.018192 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:53.436422 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.436386 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" podUID="e0657627-2d08-41e9-9326-0f7fc596f470" containerName="registry" containerID="cri-o://5fda98f02122fca83c581bc266cf7b8f796f76b4529bf7b10e8a0ffd4b4c063f" gracePeriod=30 Jul 15 13:11:53.570933 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.570906 2582 generic.go:356] "Generic (PLEG): container finished" podID="e0657627-2d08-41e9-9326-0f7fc596f470" containerID="5fda98f02122fca83c581bc266cf7b8f796f76b4529bf7b10e8a0ffd4b4c063f" exitCode=0 Jul 15 13:11:53.571060 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.570950 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" event={"ID":"e0657627-2d08-41e9-9326-0f7fc596f470","Type":"ContainerDied","Data":"5fda98f02122fca83c581bc266cf7b8f796f76b4529bf7b10e8a0ffd4b4c063f"} Jul 15 13:11:53.677438 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.677413 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:11:53.745637 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.745611 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-installation-pull-secrets\") pod \"e0657627-2d08-41e9-9326-0f7fc596f470\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " Jul 15 13:11:53.745781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.745641 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-bound-sa-token\") pod \"e0657627-2d08-41e9-9326-0f7fc596f470\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " Jul 15 13:11:53.745781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.745659 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") pod \"e0657627-2d08-41e9-9326-0f7fc596f470\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " Jul 15 13:11:53.745781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.745680 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-registry-certificates\") pod \"e0657627-2d08-41e9-9326-0f7fc596f470\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " Jul 15 13:11:53.745781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.745733 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0657627-2d08-41e9-9326-0f7fc596f470-ca-trust-extracted\") pod \"e0657627-2d08-41e9-9326-0f7fc596f470\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " Jul 15 13:11:53.745781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.745780 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-image-registry-private-configuration\") pod \"e0657627-2d08-41e9-9326-0f7fc596f470\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " Jul 15 13:11:53.746033 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.745802 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-trusted-ca\") pod \"e0657627-2d08-41e9-9326-0f7fc596f470\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " Jul 15 13:11:53.746033 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.745841 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj9gx\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-kube-api-access-cj9gx\") pod \"e0657627-2d08-41e9-9326-0f7fc596f470\" (UID: \"e0657627-2d08-41e9-9326-0f7fc596f470\") " Jul 15 13:11:53.746583 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.746291 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e0657627-2d08-41e9-9326-0f7fc596f470" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:11:53.746694 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.746672 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-registry-certificates\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:11:53.752352 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.752319 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e0657627-2d08-41e9-9326-0f7fc596f470" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:11:53.752522 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.752497 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e0657627-2d08-41e9-9326-0f7fc596f470" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:11:53.752657 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.752631 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e0657627-2d08-41e9-9326-0f7fc596f470" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:11:53.752764 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.752327 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e0657627-2d08-41e9-9326-0f7fc596f470" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:11:53.752867 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.752789 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-kube-api-access-cj9gx" (OuterVolumeSpecName: "kube-api-access-cj9gx") pod "e0657627-2d08-41e9-9326-0f7fc596f470" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470"). InnerVolumeSpecName "kube-api-access-cj9gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:11:53.753756 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.753696 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e0657627-2d08-41e9-9326-0f7fc596f470" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:11:53.757611 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.757589 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0657627-2d08-41e9-9326-0f7fc596f470-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e0657627-2d08-41e9-9326-0f7fc596f470" (UID: "e0657627-2d08-41e9-9326-0f7fc596f470"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:11:53.847195 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.847170 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj9gx\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-kube-api-access-cj9gx\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:11:53.847195 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.847192 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-installation-pull-secrets\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:11:53.847324 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.847202 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-bound-sa-token\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:11:53.847324 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.847211 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0657627-2d08-41e9-9326-0f7fc596f470-registry-tls\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:11:53.847324 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.847219 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0657627-2d08-41e9-9326-0f7fc596f470-ca-trust-extracted\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:11:53.847324 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.847227 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0657627-2d08-41e9-9326-0f7fc596f470-image-registry-private-configuration\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:11:53.847324 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:53.847236 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0657627-2d08-41e9-9326-0f7fc596f470-trusted-ca\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:11:54.575390 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:54.575352 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" event={"ID":"e0657627-2d08-41e9-9326-0f7fc596f470","Type":"ContainerDied","Data":"58433cd32bf4f453215e27aa72d56887d5a3050e9e4754543d1bdf4df71df98a"} Jul 15 13:11:54.575827 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:54.575395 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77d7b747c9-fcj99" Jul 15 13:11:54.575827 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:54.575422 2582 scope.go:117] "RemoveContainer" containerID="5fda98f02122fca83c581bc266cf7b8f796f76b4529bf7b10e8a0ffd4b4c063f" Jul 15 13:11:54.597923 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:54.597899 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77d7b747c9-fcj99"] Jul 15 13:11:54.601756 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:54.601736 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77d7b747c9-fcj99"] Jul 15 13:11:55.458891 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:55.458864 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0657627-2d08-41e9-9326-0f7fc596f470" path="/var/lib/kubelet/pods/e0657627-2d08-41e9-9326-0f7fc596f470/volumes" Jul 15 13:11:55.761863 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:55.761787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:55.894568 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:55.894540 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:57.763163 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:57.763131 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:57.949351 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:57.949323 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:58.917451 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:58.917420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:11:59.083575 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:11:59.083545 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:02.038900 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:02.038862 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" podUID="a59d4b16-44a0-41ba-89ee-ac8ef6d991ef" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 15 13:12:02.039325 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:02.038938 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" Jul 15 13:12:02.039433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:02.039401 2582 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"5bb9bfb803dae1de68bd3d6d44911efbf91166527995433ae9d1e018aec823a5"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 15 13:12:02.039483 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:02.039454 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" podUID="a59d4b16-44a0-41ba-89ee-ac8ef6d991ef" containerName="service-proxy" containerID="cri-o://5bb9bfb803dae1de68bd3d6d44911efbf91166527995433ae9d1e018aec823a5" gracePeriod=30 Jul 15 13:12:02.601193 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:02.601164 2582 generic.go:356] "Generic (PLEG): container finished" podID="a59d4b16-44a0-41ba-89ee-ac8ef6d991ef" containerID="5bb9bfb803dae1de68bd3d6d44911efbf91166527995433ae9d1e018aec823a5" exitCode=2 Jul 15 13:12:02.601344 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:02.601229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" event={"ID":"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef","Type":"ContainerDied","Data":"5bb9bfb803dae1de68bd3d6d44911efbf91166527995433ae9d1e018aec823a5"} Jul 15 13:12:02.601344 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:02.601260 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c566796b9-6cmrf" event={"ID":"a59d4b16-44a0-41ba-89ee-ac8ef6d991ef","Type":"ContainerStarted","Data":"f29120a51624d368cfc0ddb9745c34dd857b1b465a0fe2f03815a9d1f6f150dc"} Jul 15 13:12:03.790277 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:03.790247 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:03.913368 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:03.913341 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:18.398891 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:18.398811 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:39.255555 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:39.255523 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:12:39.257743 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:39.257725 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b811d4da-54d9-42e2-b09c-ada3620a5a92-metrics-certs\") pod \"network-metrics-daemon-cx6mz\" (UID: \"b811d4da-54d9-42e2-b09c-ada3620a5a92\") " pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:12:39.558517 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:39.558456 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8tgt2\"" Jul 15 13:12:39.567198 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:39.567183 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cx6mz" Jul 15 13:12:39.693799 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:39.693776 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cx6mz"] Jul 15 13:12:39.696004 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:12:39.695982 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb811d4da_54d9_42e2_b09c_ada3620a5a92.slice/crio-bf5f15f11d84861c73c3db3669464be182ae052019765e099b827fe3e735c88c WatchSource:0}: Error finding container bf5f15f11d84861c73c3db3669464be182ae052019765e099b827fe3e735c88c: Status 404 returned error can't find the container with id bf5f15f11d84861c73c3db3669464be182ae052019765e099b827fe3e735c88c Jul 15 13:12:39.719917 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:39.719893 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx6mz" event={"ID":"b811d4da-54d9-42e2-b09c-ada3620a5a92","Type":"ContainerStarted","Data":"bf5f15f11d84861c73c3db3669464be182ae052019765e099b827fe3e735c88c"} Jul 15 13:12:40.726653 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:40.726627 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx6mz" event={"ID":"b811d4da-54d9-42e2-b09c-ada3620a5a92","Type":"ContainerStarted","Data":"4ffeffb2fb0d19e0ec8f2a8b5faaf8090de68457ebf786488ee4f8a4b347838a"} Jul 15 13:12:41.732387 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:41.732349 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cx6mz" event={"ID":"b811d4da-54d9-42e2-b09c-ada3620a5a92","Type":"ContainerStarted","Data":"27e2292a87cadf6f76d7b85c10be0ded071fa3acca82b43086a48be029b542de"} Jul 15 13:12:41.753220 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:41.753172 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cx6mz" podStartSLOduration=374.813393963 podStartE2EDuration="6m15.753157519s" totalImagesPullingTime="939.763556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:12:41.751843215 +0000 UTC m=+376.921789570" watchObservedRunningTime="2026-07-15 13:12:41.753157519 +0000 UTC m=+376.923103872" Jul 15 13:12:41.891584 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:41.891560 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:41.984580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:41.984526 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:48.021728 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:48.021677 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:48.106959 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:48.106927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:51.419183 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:51.419147 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:51.513899 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:51.513855 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:12:59.955497 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:12:59.955458 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:00.039398 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:00.039365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:02.215132 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:02.215101 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:02.292434 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:02.292406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:03.785684 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:03.785652 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:03.908836 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:03.908805 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:10.625890 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:10.625856 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:32.483796 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:32.483768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:32.593907 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:32.593877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:39.905464 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:39.905431 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:13:40.008903 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:13:40.008872 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:06.643511 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:06.643416 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:07.510569 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:07.510539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:10.256677 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:10.256644 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:10.362099 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:10.362054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:13.840580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:13.840548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:13.939009 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:13.938978 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:14.688764 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:14.688737 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:14.899953 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:14.899914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:17.106725 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:17.106694 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:17.206051 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:17.206025 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:18.996293 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:18.996264 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:19.126485 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:19.126454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:22.903932 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:22.903902 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:23.107216 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.107184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:23.193418 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.193358 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:23.850384 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.850355 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dxzh9"] Jul 15 13:14:23.850575 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.850563 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0657627-2d08-41e9-9326-0f7fc596f470" containerName="registry" Jul 15 13:14:23.850620 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.850576 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0657627-2d08-41e9-9326-0f7fc596f470" containerName="registry" Jul 15 13:14:23.850654 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.850620 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0657627-2d08-41e9-9326-0f7fc596f470" containerName="registry" Jul 15 13:14:23.853428 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.853412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dxzh9" Jul 15 13:14:23.855599 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.855577 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 13:14:23.855695 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.855613 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 13:14:23.856210 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.856187 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2lvdc\"" Jul 15 13:14:23.861379 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.861356 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dxzh9"] Jul 15 13:14:23.912835 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.912811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e67630a-4751-4c56-9adb-594d9a37ba1c-bound-sa-token\") pod \"cert-manager-55b4d954d5-dxzh9\" (UID: \"3e67630a-4751-4c56-9adb-594d9a37ba1c\") " pod="cert-manager/cert-manager-55b4d954d5-dxzh9" Jul 15 13:14:23.913117 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:23.912838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gjc\" (UniqueName: \"kubernetes.io/projected/3e67630a-4751-4c56-9adb-594d9a37ba1c-kube-api-access-k7gjc\") pod \"cert-manager-55b4d954d5-dxzh9\" (UID: \"3e67630a-4751-4c56-9adb-594d9a37ba1c\") " pod="cert-manager/cert-manager-55b4d954d5-dxzh9" Jul 15 13:14:24.013634 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:24.013612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gjc\" (UniqueName: \"kubernetes.io/projected/3e67630a-4751-4c56-9adb-594d9a37ba1c-kube-api-access-k7gjc\") pod \"cert-manager-55b4d954d5-dxzh9\" (UID: \"3e67630a-4751-4c56-9adb-594d9a37ba1c\") " pod="cert-manager/cert-manager-55b4d954d5-dxzh9" Jul 15 13:14:24.013761 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:24.013662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e67630a-4751-4c56-9adb-594d9a37ba1c-bound-sa-token\") pod \"cert-manager-55b4d954d5-dxzh9\" (UID: \"3e67630a-4751-4c56-9adb-594d9a37ba1c\") " pod="cert-manager/cert-manager-55b4d954d5-dxzh9" Jul 15 13:14:24.023192 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:24.023163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gjc\" (UniqueName: \"kubernetes.io/projected/3e67630a-4751-4c56-9adb-594d9a37ba1c-kube-api-access-k7gjc\") pod \"cert-manager-55b4d954d5-dxzh9\" (UID: \"3e67630a-4751-4c56-9adb-594d9a37ba1c\") " pod="cert-manager/cert-manager-55b4d954d5-dxzh9" Jul 15 13:14:24.023292 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:24.023259 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3e67630a-4751-4c56-9adb-594d9a37ba1c-bound-sa-token\") pod \"cert-manager-55b4d954d5-dxzh9\" (UID: \"3e67630a-4751-4c56-9adb-594d9a37ba1c\") " pod="cert-manager/cert-manager-55b4d954d5-dxzh9" Jul 15 13:14:24.162435 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:24.162386 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dxzh9" Jul 15 13:14:24.283692 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:24.283662 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dxzh9"] Jul 15 13:14:24.287345 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:14:24.287318 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e67630a_4751_4c56_9adb_594d9a37ba1c.slice/crio-12f111db20b10565a314810bc2a2d0f2f26809b60cbee18b3361bd4e5e4b482d WatchSource:0}: Error finding container 12f111db20b10565a314810bc2a2d0f2f26809b60cbee18b3361bd4e5e4b482d: Status 404 returned error can't find the container with id 12f111db20b10565a314810bc2a2d0f2f26809b60cbee18b3361bd4e5e4b482d Jul 15 13:14:25.066195 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.066150 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dxzh9" event={"ID":"3e67630a-4751-4c56-9adb-594d9a37ba1c","Type":"ContainerStarted","Data":"12f111db20b10565a314810bc2a2d0f2f26809b60cbee18b3361bd4e5e4b482d"} Jul 15 13:14:25.287830 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.287796 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx"] Jul 15 13:14:25.290825 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.290800 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" Jul 15 13:14:25.292997 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.292825 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 13:14:25.293425 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.293401 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:14:25.293513 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.293401 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hd8n8\"" Jul 15 13:14:25.302384 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.302365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx"] Jul 15 13:14:25.423642 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.423563 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59bh\" (UniqueName: \"kubernetes.io/projected/ca5e3b86-2247-4e9b-8515-2926c8aaf0de-kube-api-access-c59bh\") pod \"openshift-lws-operator-fd8ccff4c-8j9cx\" (UID: \"ca5e3b86-2247-4e9b-8515-2926c8aaf0de\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" Jul 15 13:14:25.423642 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.423607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca5e3b86-2247-4e9b-8515-2926c8aaf0de-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8j9cx\" (UID: \"ca5e3b86-2247-4e9b-8515-2926c8aaf0de\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" Jul 15 13:14:25.524639 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.524602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59bh\" (UniqueName: \"kubernetes.io/projected/ca5e3b86-2247-4e9b-8515-2926c8aaf0de-kube-api-access-c59bh\") pod \"openshift-lws-operator-fd8ccff4c-8j9cx\" (UID: \"ca5e3b86-2247-4e9b-8515-2926c8aaf0de\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" Jul 15 13:14:25.524639 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.524647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca5e3b86-2247-4e9b-8515-2926c8aaf0de-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8j9cx\" (UID: \"ca5e3b86-2247-4e9b-8515-2926c8aaf0de\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" Jul 15 13:14:25.525047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.525027 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca5e3b86-2247-4e9b-8515-2926c8aaf0de-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8j9cx\" (UID: \"ca5e3b86-2247-4e9b-8515-2926c8aaf0de\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" Jul 15 13:14:25.532993 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.532959 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59bh\" (UniqueName: \"kubernetes.io/projected/ca5e3b86-2247-4e9b-8515-2926c8aaf0de-kube-api-access-c59bh\") pod \"openshift-lws-operator-fd8ccff4c-8j9cx\" (UID: \"ca5e3b86-2247-4e9b-8515-2926c8aaf0de\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" Jul 15 13:14:25.603008 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.602977 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" Jul 15 13:14:25.826221 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.826185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:25.929176 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:25.929139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:27.137055 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:27.136992 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx"] Jul 15 13:14:27.141685 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:14:27.141663 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5e3b86_2247_4e9b_8515_2926c8aaf0de.slice/crio-bb512684cb97516dd038f95cf4764d8382276112a47380df6fea22f117cf8247 WatchSource:0}: Error finding container bb512684cb97516dd038f95cf4764d8382276112a47380df6fea22f117cf8247: Status 404 returned error can't find the container with id bb512684cb97516dd038f95cf4764d8382276112a47380df6fea22f117cf8247 Jul 15 13:14:28.079360 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:28.079322 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dxzh9" event={"ID":"3e67630a-4751-4c56-9adb-594d9a37ba1c","Type":"ContainerStarted","Data":"7d9c397284b451ee6c3d14810fbdbd7887f6b5041c07c845cef05d6c1127f101"} Jul 15 13:14:28.080659 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:28.080627 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" event={"ID":"ca5e3b86-2247-4e9b-8515-2926c8aaf0de","Type":"ContainerStarted","Data":"bb512684cb97516dd038f95cf4764d8382276112a47380df6fea22f117cf8247"} Jul 15 13:14:28.098465 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:28.098416 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dxzh9" podStartSLOduration=2.309359414 podStartE2EDuration="5.098401466s" totalImagesPullingTime="2.789042052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:14:28.097191215 +0000 UTC m=+483.267137569" watchObservedRunningTime="2026-07-15 13:14:28.098401466 +0000 UTC m=+483.268347819" Jul 15 13:14:32.095781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:32.095747 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" event={"ID":"ca5e3b86-2247-4e9b-8515-2926c8aaf0de","Type":"ContainerStarted","Data":"c7c48059706172035185bdff725b3b22bea7f1e78a7594e2cc7d947c4d744a35"} Jul 15 13:14:32.119882 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:32.119841 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8j9cx" podStartSLOduration=2.878130604 podStartE2EDuration="7.11982586s" totalImagesPullingTime="4.241695256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:14:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:14:32.117169913 +0000 UTC m=+487.287116265" watchObservedRunningTime="2026-07-15 13:14:32.11982586 +0000 UTC m=+487.289772212" Jul 15 13:14:32.842186 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:32.842152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:32.946050 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:32.946016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:34.342161 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:34.342131 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:34.466496 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:34.466466 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:34.669113 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:34.669034 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:46.470323 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:46.470292 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:46.565623 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:46.565593 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:47.103996 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.103962 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:47.345995 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.345960 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:47.474252 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.474227 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:47.517072 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.517046 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq"] Jul 15 13:14:47.523463 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.523446 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.528801 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.528784 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 13:14:47.530914 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.530890 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 13:14:47.531183 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.531169 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 13:14:47.531283 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.531200 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-snv8d\"" Jul 15 13:14:47.531283 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.531209 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 13:14:47.565988 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.565969 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-manifests\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.566081 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.566004 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-tmp\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.566161 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.566095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.566161 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.566138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-webhook-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.566363 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.566163 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrnp5\" (UniqueName: \"kubernetes.io/projected/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-kube-api-access-zrnp5\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.567744 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.567707 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq"] Jul 15 13:14:47.667233 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.667205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.667370 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.667244 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-webhook-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.667370 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.667271 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrnp5\" (UniqueName: \"kubernetes.io/projected/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-kube-api-access-zrnp5\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.667370 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.667299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-manifests\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.667370 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.667333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-tmp\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.667754 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.667707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-manifests\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.669449 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.669423 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-tmp\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.669575 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.669557 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.669746 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.669709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-webhook-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.674930 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.674908 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrnp5\" (UniqueName: \"kubernetes.io/projected/e339bef8-5ebf-4aad-b81c-a54ca8c1d23d-kube-api-access-zrnp5\") pod \"opendatahub-operator-controller-manager-869c48857d-5w6tq\" (UID: \"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.696629 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.696607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:47.833071 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.833019 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:47.848066 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.848042 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:47.933723 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.933674 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:47.967884 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:47.967848 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq"] Jul 15 13:14:47.971184 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:14:47.971153 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode339bef8_5ebf_4aad_b81c_a54ca8c1d23d.slice/crio-aa58709084fe36a63f8811e5163173b8ab205463cd34cb6e77fc9ae420c04c8a WatchSource:0}: Error finding container aa58709084fe36a63f8811e5163173b8ab205463cd34cb6e77fc9ae420c04c8a: Status 404 returned error can't find the container with id aa58709084fe36a63f8811e5163173b8ab205463cd34cb6e77fc9ae420c04c8a Jul 15 13:14:48.008348 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:48.008319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:48.108787 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:48.108733 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:48.145848 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:48.145828 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" event={"ID":"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d","Type":"ContainerStarted","Data":"aa58709084fe36a63f8811e5163173b8ab205463cd34cb6e77fc9ae420c04c8a"} Jul 15 13:14:48.211110 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:48.211087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:52.953402 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:52.953376 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:53.068287 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:53.068257 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:14:53.165831 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:53.165760 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" event={"ID":"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d","Type":"ContainerStarted","Data":"6489eea31a90184e9f3ad6718d95af8f50998f6184b92d9fb825832fe8b82f08"} Jul 15 13:14:54.171154 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:54.171123 2582 generic.go:356] "Generic (PLEG): container finished" podID="e339bef8-5ebf-4aad-b81c-a54ca8c1d23d" containerID="6489eea31a90184e9f3ad6718d95af8f50998f6184b92d9fb825832fe8b82f08" exitCode=0 Jul 15 13:14:54.171505 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:54.171206 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" event={"ID":"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d","Type":"ContainerDied","Data":"6489eea31a90184e9f3ad6718d95af8f50998f6184b92d9fb825832fe8b82f08"} Jul 15 13:14:55.176186 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:55.176149 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" event={"ID":"e339bef8-5ebf-4aad-b81c-a54ca8c1d23d","Type":"ContainerStarted","Data":"a89f044353445c4700050a8e09bd3c89ca9995b845c905a05f873bbd3b992077"} Jul 15 13:14:55.176575 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:55.176275 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:14:55.203560 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:14:55.203512 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" podStartSLOduration=1.741538292 podStartE2EDuration="8.20349797s" totalImagesPullingTime="5.400008641s" totalInitContainerRuntime="1.061951037s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:14:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:14:55.202970852 +0000 UTC m=+510.372917215" watchObservedRunningTime="2026-07-15 13:14:55.20349797 +0000 UTC m=+510.373444304" Jul 15 13:15:06.182002 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:06.181969 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-5w6tq" Jul 15 13:15:09.079309 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.079258 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:09.221147 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.221104 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:09.813521 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.813488 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vs5qs"] Jul 15 13:15:09.821351 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.821333 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:09.823317 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.823297 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 13:15:09.823439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.823424 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2jvnx\"" Jul 15 13:15:09.828838 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.828810 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vs5qs"] Jul 15 13:15:09.925979 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.925938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dncxw\" (UniqueName: \"kubernetes.io/projected/d5ec325f-c272-40f8-a9dc-2b55642f757d-kube-api-access-dncxw\") pod \"odh-model-controller-7998c67dc4-vs5qs\" (UID: \"d5ec325f-c272-40f8-a9dc-2b55642f757d\") " pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:09.925979 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:09.925986 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ec325f-c272-40f8-a9dc-2b55642f757d-cert\") pod \"odh-model-controller-7998c67dc4-vs5qs\" (UID: \"d5ec325f-c272-40f8-a9dc-2b55642f757d\") " pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:10.027240 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.027202 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dncxw\" (UniqueName: \"kubernetes.io/projected/d5ec325f-c272-40f8-a9dc-2b55642f757d-kube-api-access-dncxw\") pod \"odh-model-controller-7998c67dc4-vs5qs\" (UID: \"d5ec325f-c272-40f8-a9dc-2b55642f757d\") " pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:10.027413 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.027257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ec325f-c272-40f8-a9dc-2b55642f757d-cert\") pod \"odh-model-controller-7998c67dc4-vs5qs\" (UID: \"d5ec325f-c272-40f8-a9dc-2b55642f757d\") " pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:10.027413 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:10.027371 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 13:15:10.027503 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:10.027443 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ec325f-c272-40f8-a9dc-2b55642f757d-cert podName:d5ec325f-c272-40f8-a9dc-2b55642f757d nodeName:}" failed. No retries permitted until 2026-07-15 13:15:10.527422536 +0000 UTC m=+525.697368872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5ec325f-c272-40f8-a9dc-2b55642f757d-cert") pod "odh-model-controller-7998c67dc4-vs5qs" (UID: "d5ec325f-c272-40f8-a9dc-2b55642f757d") : secret "odh-model-controller-webhook-cert" not found Jul 15 13:15:10.035534 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.035511 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dncxw\" (UniqueName: \"kubernetes.io/projected/d5ec325f-c272-40f8-a9dc-2b55642f757d-kube-api-access-dncxw\") pod \"odh-model-controller-7998c67dc4-vs5qs\" (UID: \"d5ec325f-c272-40f8-a9dc-2b55642f757d\") " pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:10.412593 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.412560 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rp7bt"] Jul 15 13:15:10.415479 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.415463 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:10.417393 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.417373 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 13:15:10.417702 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.417496 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rtc8g\"" Jul 15 13:15:10.426150 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.426124 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rp7bt"] Jul 15 13:15:10.533068 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.533031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ec325f-c272-40f8-a9dc-2b55642f757d-cert\") pod \"odh-model-controller-7998c67dc4-vs5qs\" (UID: \"d5ec325f-c272-40f8-a9dc-2b55642f757d\") " pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:10.533234 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.533103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-tls-certs\") pod \"model-serving-api-54d89bd5fc-rp7bt\" (UID: \"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:10.533234 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.533138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgzqm\" (UniqueName: \"kubernetes.io/projected/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-kube-api-access-cgzqm\") pod \"model-serving-api-54d89bd5fc-rp7bt\" (UID: \"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:10.535387 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.535367 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ec325f-c272-40f8-a9dc-2b55642f757d-cert\") pod \"odh-model-controller-7998c67dc4-vs5qs\" (UID: \"d5ec325f-c272-40f8-a9dc-2b55642f757d\") " pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:10.634023 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.633992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-tls-certs\") pod \"model-serving-api-54d89bd5fc-rp7bt\" (UID: \"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:10.634181 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.634033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgzqm\" (UniqueName: \"kubernetes.io/projected/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-kube-api-access-cgzqm\") pod \"model-serving-api-54d89bd5fc-rp7bt\" (UID: \"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:10.634181 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:10.634153 2582 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 15 13:15:10.634247 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:10.634221 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-tls-certs podName:285afc68-cca3-4543-b1f7-dd5e0d7e1f2e nodeName:}" failed. No retries permitted until 2026-07-15 13:15:11.134201795 +0000 UTC m=+526.304148141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-tls-certs") pod "model-serving-api-54d89bd5fc-rp7bt" (UID: "285afc68-cca3-4543-b1f7-dd5e0d7e1f2e") : secret "model-serving-api-tls" not found Jul 15 13:15:10.642492 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.642461 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgzqm\" (UniqueName: \"kubernetes.io/projected/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-kube-api-access-cgzqm\") pod \"model-serving-api-54d89bd5fc-rp7bt\" (UID: \"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:10.732603 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.732572 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:10.864064 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:10.864030 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vs5qs"] Jul 15 13:15:10.866543 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:15:10.866513 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ec325f_c272_40f8_a9dc_2b55642f757d.slice/crio-59a4824ff54f2a096c76ac7dcbd5a7d4810769ffecdea921a83734a5083a899e WatchSource:0}: Error finding container 59a4824ff54f2a096c76ac7dcbd5a7d4810769ffecdea921a83734a5083a899e: Status 404 returned error can't find the container with id 59a4824ff54f2a096c76ac7dcbd5a7d4810769ffecdea921a83734a5083a899e Jul 15 13:15:11.137113 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.137022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-tls-certs\") pod \"model-serving-api-54d89bd5fc-rp7bt\" (UID: \"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:11.139354 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.139327 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285afc68-cca3-4543-b1f7-dd5e0d7e1f2e-tls-certs\") pod \"model-serving-api-54d89bd5fc-rp7bt\" (UID: \"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:11.230651 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.230610 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" event={"ID":"d5ec325f-c272-40f8-a9dc-2b55642f757d","Type":"ContainerStarted","Data":"59a4824ff54f2a096c76ac7dcbd5a7d4810769ffecdea921a83734a5083a899e"} Jul 15 13:15:11.264252 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.264222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:11.324595 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.324557 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd"] Jul 15 13:15:11.325433 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.325406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:11.329674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.329657 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.331856 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.331831 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 13:15:11.333278 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.333260 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 13:15:11.333569 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.333549 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 13:15:11.333642 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.333556 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 13:15:11.349518 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.349486 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd"] Jul 15 13:15:11.355406 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.355375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:11.439255 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.439217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9051a7d5-2c0f-46d8-834f-8413e7996134-tls-certs\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.439598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.439266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn55j\" (UniqueName: \"kubernetes.io/projected/9051a7d5-2c0f-46d8-834f-8413e7996134-kube-api-access-dn55j\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.439598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.439300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9051a7d5-2c0f-46d8-834f-8413e7996134-tmp\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.461683 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.461653 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rp7bt"] Jul 15 13:15:11.465553 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:15:11.465526 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285afc68_cca3_4543_b1f7_dd5e0d7e1f2e.slice/crio-d56e9e44ad162c83a960cdf7871a17a59039bab20b984606c88f15f35dbe8855 WatchSource:0}: Error finding container d56e9e44ad162c83a960cdf7871a17a59039bab20b984606c88f15f35dbe8855: Status 404 returned error can't find the container with id d56e9e44ad162c83a960cdf7871a17a59039bab20b984606c88f15f35dbe8855 Jul 15 13:15:11.540529 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.540490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9051a7d5-2c0f-46d8-834f-8413e7996134-tls-certs\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.540735 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.540553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn55j\" (UniqueName: \"kubernetes.io/projected/9051a7d5-2c0f-46d8-834f-8413e7996134-kube-api-access-dn55j\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.540735 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.540597 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9051a7d5-2c0f-46d8-834f-8413e7996134-tmp\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.543266 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.543231 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9051a7d5-2c0f-46d8-834f-8413e7996134-tmp\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.543459 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.543437 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9051a7d5-2c0f-46d8-834f-8413e7996134-tls-certs\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.549860 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.549837 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn55j\" (UniqueName: \"kubernetes.io/projected/9051a7d5-2c0f-46d8-834f-8413e7996134-kube-api-access-dn55j\") pod \"kube-auth-proxy-6b6cfc997-56nkd\" (UID: \"9051a7d5-2c0f-46d8-834f-8413e7996134\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.653542 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.653506 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" Jul 15 13:15:11.818269 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:11.818237 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd"] Jul 15 13:15:11.821731 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:15:11.821684 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9051a7d5_2c0f_46d8_834f_8413e7996134.slice/crio-c54582256a5a426ab0572892fd524b879d1cb24ab9acb008776072cc72405617 WatchSource:0}: Error finding container c54582256a5a426ab0572892fd524b879d1cb24ab9acb008776072cc72405617: Status 404 returned error can't find the container with id c54582256a5a426ab0572892fd524b879d1cb24ab9acb008776072cc72405617 Jul 15 13:15:12.238003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.237963 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" event={"ID":"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e","Type":"ContainerStarted","Data":"d56e9e44ad162c83a960cdf7871a17a59039bab20b984606c88f15f35dbe8855"} Jul 15 13:15:12.239567 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.239538 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" event={"ID":"9051a7d5-2c0f-46d8-834f-8413e7996134","Type":"ContainerStarted","Data":"c54582256a5a426ab0572892fd524b879d1cb24ab9acb008776072cc72405617"} Jul 15 13:15:12.712001 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.711955 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:12.725744 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.725685 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn"] Jul 15 13:15:12.734307 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.734272 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.736392 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.736365 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 13:15:12.736533 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.736365 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 13:15:12.736782 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.736762 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 13:15:12.736896 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.736815 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gs4ls\"" Jul 15 13:15:12.737007 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.736990 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 13:15:12.746663 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.746636 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn"] Jul 15 13:15:12.796491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.796458 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:12.851276 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.851229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-istio-token\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.851477 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.851284 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-cacerts\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.851477 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.851356 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.851477 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.851401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.851477 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.851463 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.851693 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.851494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c1f218-a462-4225-a443-69a3d29d5087-local-certs\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.851693 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.851574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfv6h\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-kube-api-access-nfv6h\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.952871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.952837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-istio-token\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.952871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.952878 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-cacerts\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.953101 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.952902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.953182 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.953156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.953240 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.953210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.953240 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.953232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c1f218-a462-4225-a443-69a3d29d5087-local-certs\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.953359 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.953296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfv6h\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-kube-api-access-nfv6h\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.953910 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.953883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.955538 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.955513 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.955729 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.955690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.955804 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.955745 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-cacerts\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.955863 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.955807 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c1f218-a462-4225-a443-69a3d29d5087-local-certs\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.961284 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.961262 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-istio-token\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:12.961390 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:12.961281 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfv6h\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-kube-api-access-nfv6h\") pod \"istiod-openshift-gateway-7574459656-mgxpn\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:13.048032 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:13.047950 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:13.745033 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:13.744988 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:14.389769 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:14.389740 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn"] Jul 15 13:15:14.392948 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:15:14.392916 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c1f218_a462_4225_a443_69a3d29d5087.slice/crio-a612f8b741380616dd5e1d2affd317206fa9b2d2e04fea5501e08e986f6ffc52 WatchSource:0}: Error finding container a612f8b741380616dd5e1d2affd317206fa9b2d2e04fea5501e08e986f6ffc52: Status 404 returned error can't find the container with id a612f8b741380616dd5e1d2affd317206fa9b2d2e04fea5501e08e986f6ffc52 Jul 15 13:15:15.255706 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:15.255666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" event={"ID":"285afc68-cca3-4543-b1f7-dd5e0d7e1f2e","Type":"ContainerStarted","Data":"55253a1c034f198940b51921314a9e5c3c6af85fae8d2199b102ebce667d18db"} Jul 15 13:15:15.256183 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:15.255774 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:15.257182 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:15.257148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" event={"ID":"00c1f218-a462-4225-a443-69a3d29d5087","Type":"ContainerStarted","Data":"a612f8b741380616dd5e1d2affd317206fa9b2d2e04fea5501e08e986f6ffc52"} Jul 15 13:15:15.258859 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:15.258825 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" event={"ID":"d5ec325f-c272-40f8-a9dc-2b55642f757d","Type":"ContainerStarted","Data":"e866b919415bfd2ca22f304eb1cdbbb9d8c5db9e47618b1e2e1e3741f87a308e"} Jul 15 13:15:15.258961 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:15.258891 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:15.276485 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:15.276428 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" podStartSLOduration=2.521541918 podStartE2EDuration="5.27641263s" totalImagesPullingTime="2.754870712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:15.275204733 +0000 UTC m=+530.445151112" watchObservedRunningTime="2026-07-15 13:15:15.27641263 +0000 UTC m=+530.446358967" Jul 15 13:15:15.292638 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:15.292586 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" podStartSLOduration=2.923211933 podStartE2EDuration="6.292567727s" totalImagesPullingTime="3.369355794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:15.291535826 +0000 UTC m=+530.461482179" watchObservedRunningTime="2026-07-15 13:15:15.292567727 +0000 UTC m=+530.462514079" Jul 15 13:15:16.264765 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:16.264729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" event={"ID":"9051a7d5-2c0f-46d8-834f-8413e7996134","Type":"ContainerStarted","Data":"8c6e8aec98975d18ab7e7b60f5372d5d3e4eceed0fda30ef4324a0bff8c945ac"} Jul 15 13:15:16.287225 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:16.287145 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-56nkd" podStartSLOduration=1.741398709 podStartE2EDuration="5.287124756s" totalImagesPullingTime="3.545726047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:16.287020134 +0000 UTC m=+531.456966508" watchObservedRunningTime="2026-07-15 13:15:16.287124756 +0000 UTC m=+531.457071155" Jul 15 13:15:17.087151 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:17.087116 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:15:17.087280 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:17.087178 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:15:17.270746 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:17.270700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" event={"ID":"00c1f218-a462-4225-a443-69a3d29d5087","Type":"ContainerStarted","Data":"367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a"} Jul 15 13:15:17.271131 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:17.271018 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:17.328353 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:17.328303 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" podStartSLOduration=2.636742885 podStartE2EDuration="5.328287031s" totalImagesPullingTime="2.691544146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:17.326699384 +0000 UTC m=+532.496645737" watchObservedRunningTime="2026-07-15 13:15:17.328287031 +0000 UTC m=+532.498233384" Jul 15 13:15:18.277016 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:18.276981 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-mgxpn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 13:15:18.277507 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:18.277034 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" podUID="00c1f218-a462-4225-a443-69a3d29d5087" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:15:18.780942 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:18.780896 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:18.889641 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:18.889602 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:19.703762 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.703729 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s"] Jul 15 13:15:19.706942 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.706926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.710027 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.710012 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pndhb\"" Jul 15 13:15:19.727852 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.727829 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s"] Jul 15 13:15:19.752471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.752449 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:19.808092 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808069 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbpv8\" (UniqueName: \"kubernetes.io/projected/bc762472-b253-4d72-a6c1-6635a0b10a39-kube-api-access-wbpv8\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.808184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.808184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.808184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.808184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808155 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.808314 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.808314 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.808314 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.808405 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.808352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc762472-b253-4d72-a6c1-6635a0b10a39-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909290 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909382 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909382 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc762472-b253-4d72-a6c1-6635a0b10a39-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909382 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbpv8\" (UniqueName: \"kubernetes.io/projected/bc762472-b253-4d72-a6c1-6635a0b10a39-kube-api-access-wbpv8\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909496 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909552 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909602 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909650 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909705 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909687 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909775 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909837 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909775 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.909977 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.909954 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.910069 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.910001 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.910197 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.910176 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc762472-b253-4d72-a6c1-6635a0b10a39-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.911828 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.911810 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.911921 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.911828 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.919582 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.919564 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc762472-b253-4d72-a6c1-6635a0b10a39-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.920154 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.920132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbpv8\" (UniqueName: \"kubernetes.io/projected/bc762472-b253-4d72-a6c1-6635a0b10a39-kube-api-access-wbpv8\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678v6d2s\" (UID: \"bc762472-b253-4d72-a6c1-6635a0b10a39\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:19.941482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:19.941460 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:20.016590 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.016534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:20.095471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.095440 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:20.156836 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.156811 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s"] Jul 15 13:15:20.159231 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:15:20.159204 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc762472_b253_4d72_a6c1_6635a0b10a39.slice/crio-d8f0476a744fe7bb58dfd1186814f49e54deea08a5e656e321dee6480b4eded0 WatchSource:0}: Error finding container d8f0476a744fe7bb58dfd1186814f49e54deea08a5e656e321dee6480b4eded0: Status 404 returned error can't find the container with id d8f0476a744fe7bb58dfd1186814f49e54deea08a5e656e321dee6480b4eded0 Jul 15 13:15:20.285983 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.285895 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" event={"ID":"bc762472-b253-4d72-a6c1-6635a0b10a39","Type":"ContainerStarted","Data":"d8f0476a744fe7bb58dfd1186814f49e54deea08a5e656e321dee6480b4eded0"} Jul 15 13:15:20.558171 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.558081 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:20.648624 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.648593 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9bswr"] Jul 15 13:15:20.649676 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.649641 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:20.652009 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.651982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:20.654345 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.654318 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 13:15:20.654462 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.654375 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-v567n\"" Jul 15 13:15:20.658944 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.658793 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x"] Jul 15 13:15:20.662148 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.662126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:20.664805 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.664763 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sj8gd\"" Jul 15 13:15:20.665138 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.665115 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 13:15:20.666616 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.666587 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9bswr"] Jul 15 13:15:20.697496 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.697456 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x"] Jul 15 13:15:20.746960 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.746931 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:20.816057 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.815977 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4t8v\" (UniqueName: \"kubernetes.io/projected/216f8da5-74c0-425a-ba2c-e49228fc7295-kube-api-access-p4t8v\") pod \"llmisvc-controller-manager-777c6bd7d5-7lz7x\" (UID: \"216f8da5-74c0-425a-ba2c-e49228fc7295\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:20.816057 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.816045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/786c6d59-4d31-48a3-a4e7-0c73d768d911-cert\") pod \"kserve-controller-manager-d6f4b65b8-9bswr\" (UID: \"786c6d59-4d31-48a3-a4e7-0c73d768d911\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:20.816235 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.816126 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7z2s\" (UniqueName: \"kubernetes.io/projected/786c6d59-4d31-48a3-a4e7-0c73d768d911-kube-api-access-s7z2s\") pod \"kserve-controller-manager-d6f4b65b8-9bswr\" (UID: \"786c6d59-4d31-48a3-a4e7-0c73d768d911\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:20.816235 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.816174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216f8da5-74c0-425a-ba2c-e49228fc7295-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7lz7x\" (UID: \"216f8da5-74c0-425a-ba2c-e49228fc7295\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:20.917437 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.917400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4t8v\" (UniqueName: \"kubernetes.io/projected/216f8da5-74c0-425a-ba2c-e49228fc7295-kube-api-access-p4t8v\") pod \"llmisvc-controller-manager-777c6bd7d5-7lz7x\" (UID: \"216f8da5-74c0-425a-ba2c-e49228fc7295\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:20.917751 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.917603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/786c6d59-4d31-48a3-a4e7-0c73d768d911-cert\") pod \"kserve-controller-manager-d6f4b65b8-9bswr\" (UID: \"786c6d59-4d31-48a3-a4e7-0c73d768d911\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:20.918323 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:20.917938 2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 15 13:15:20.918323 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.917971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7z2s\" (UniqueName: \"kubernetes.io/projected/786c6d59-4d31-48a3-a4e7-0c73d768d911-kube-api-access-s7z2s\") pod \"kserve-controller-manager-d6f4b65b8-9bswr\" (UID: \"786c6d59-4d31-48a3-a4e7-0c73d768d911\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:20.918323 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:20.918010 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786c6d59-4d31-48a3-a4e7-0c73d768d911-cert podName:786c6d59-4d31-48a3-a4e7-0c73d768d911 nodeName:}" failed. No retries permitted until 2026-07-15 13:15:21.417986922 +0000 UTC m=+536.587933255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/786c6d59-4d31-48a3-a4e7-0c73d768d911-cert") pod "kserve-controller-manager-d6f4b65b8-9bswr" (UID: "786c6d59-4d31-48a3-a4e7-0c73d768d911") : secret "kserve-webhook-server-cert" not found Jul 15 13:15:20.918323 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.918050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216f8da5-74c0-425a-ba2c-e49228fc7295-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7lz7x\" (UID: \"216f8da5-74c0-425a-ba2c-e49228fc7295\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:20.918323 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:20.918152 2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 15 13:15:20.918323 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:20.918197 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/216f8da5-74c0-425a-ba2c-e49228fc7295-cert podName:216f8da5-74c0-425a-ba2c-e49228fc7295 nodeName:}" failed. No retries permitted until 2026-07-15 13:15:21.418180564 +0000 UTC m=+536.588126910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/216f8da5-74c0-425a-ba2c-e49228fc7295-cert") pod "llmisvc-controller-manager-777c6bd7d5-7lz7x" (UID: "216f8da5-74c0-425a-ba2c-e49228fc7295") : secret "llmisvc-webhook-server-cert" not found Jul 15 13:15:20.934890 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.934866 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7z2s\" (UniqueName: \"kubernetes.io/projected/786c6d59-4d31-48a3-a4e7-0c73d768d911-kube-api-access-s7z2s\") pod \"kserve-controller-manager-d6f4b65b8-9bswr\" (UID: \"786c6d59-4d31-48a3-a4e7-0c73d768d911\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:20.944383 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.944358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4t8v\" (UniqueName: \"kubernetes.io/projected/216f8da5-74c0-425a-ba2c-e49228fc7295-kube-api-access-p4t8v\") pod \"llmisvc-controller-manager-777c6bd7d5-7lz7x\" (UID: \"216f8da5-74c0-425a-ba2c-e49228fc7295\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:20.996228 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:20.996188 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:21.276435 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.276406 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:21.419703 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.419672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:21.422819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.422796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/786c6d59-4d31-48a3-a4e7-0c73d768d911-cert\") pod \"kserve-controller-manager-d6f4b65b8-9bswr\" (UID: \"786c6d59-4d31-48a3-a4e7-0c73d768d911\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:21.422941 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.422890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216f8da5-74c0-425a-ba2c-e49228fc7295-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7lz7x\" (UID: \"216f8da5-74c0-425a-ba2c-e49228fc7295\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:21.425201 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.425178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/786c6d59-4d31-48a3-a4e7-0c73d768d911-cert\") pod \"kserve-controller-manager-d6f4b65b8-9bswr\" (UID: \"786c6d59-4d31-48a3-a4e7-0c73d768d911\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:21.425305 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.425264 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216f8da5-74c0-425a-ba2c-e49228fc7295-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7lz7x\" (UID: \"216f8da5-74c0-425a-ba2c-e49228fc7295\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:21.564332 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.564239 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:21.574753 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.574728 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:21.880736 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.880637 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:21.994707 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:21.994669 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:22.101731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:22.101683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:22.443498 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:22.443468 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:15:22.443627 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:22.443550 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:15:22.443627 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:22.443602 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:15:22.513310 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:22.513264 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x"] Jul 15 13:15:22.516847 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:15:22.516794 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod216f8da5_74c0_425a_ba2c_e49228fc7295.slice/crio-9dcd66df76404516e8e0d5abf0de3791052918f17f9ae62973544a32e8efb8d9 WatchSource:0}: Error finding container 9dcd66df76404516e8e0d5abf0de3791052918f17f9ae62973544a32e8efb8d9: Status 404 returned error can't find the container with id 9dcd66df76404516e8e0d5abf0de3791052918f17f9ae62973544a32e8efb8d9 Jul 15 13:15:22.533085 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:22.533060 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9bswr"] Jul 15 13:15:22.535566 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:15:22.535542 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786c6d59_4d31_48a3_a4e7_0c73d768d911.slice/crio-fb6280f6659f233e80ffa0e1f10a3cbcc1ccc5694d0bc28d9dae23bf593e7be1 WatchSource:0}: Error finding container fb6280f6659f233e80ffa0e1f10a3cbcc1ccc5694d0bc28d9dae23bf593e7be1: Status 404 returned error can't find the container with id fb6280f6659f233e80ffa0e1f10a3cbcc1ccc5694d0bc28d9dae23bf593e7be1 Jul 15 13:15:23.304985 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:23.304949 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" event={"ID":"216f8da5-74c0-425a-ba2c-e49228fc7295","Type":"ContainerStarted","Data":"9dcd66df76404516e8e0d5abf0de3791052918f17f9ae62973544a32e8efb8d9"} Jul 15 13:15:23.306478 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:23.306451 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" event={"ID":"786c6d59-4d31-48a3-a4e7-0c73d768d911","Type":"ContainerStarted","Data":"fb6280f6659f233e80ffa0e1f10a3cbcc1ccc5694d0bc28d9dae23bf593e7be1"} Jul 15 13:15:23.308077 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:23.308052 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" event={"ID":"bc762472-b253-4d72-a6c1-6635a0b10a39","Type":"ContainerStarted","Data":"a7a1f1f6e6d360c1aa1740023261ada43e3f1ed204d9d2f77f53f82356050edf"} Jul 15 13:15:23.346775 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:23.346703 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" podStartSLOduration=2.064738455 podStartE2EDuration="4.34668886s" totalImagesPullingTime="2.281950405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:23.344982648 +0000 UTC m=+538.514929002" watchObservedRunningTime="2026-07-15 13:15:23.34668886 +0000 UTC m=+538.516635212" Jul 15 13:15:23.644336 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:23.644259 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn"] Jul 15 13:15:23.644508 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:23.644482 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" podUID="00c1f218-a462-4225-a443-69a3d29d5087" containerName="discovery" containerID="cri-o://367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a" gracePeriod=30 Jul 15 13:15:23.898835 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:23.898782 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:24.016899 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.016872 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:24.021670 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.021645 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:24.044624 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.044599 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-kubeconfig\") pod \"00c1f218-a462-4225-a443-69a3d29d5087\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " Jul 15 13:15:24.044761 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.044652 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-ca-configmap\") pod \"00c1f218-a462-4225-a443-69a3d29d5087\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " Jul 15 13:15:24.044761 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.044686 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-cacerts\") pod \"00c1f218-a462-4225-a443-69a3d29d5087\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " Jul 15 13:15:24.044761 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.044743 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c1f218-a462-4225-a443-69a3d29d5087-local-certs\") pod \"00c1f218-a462-4225-a443-69a3d29d5087\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " Jul 15 13:15:24.044937 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.044778 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfv6h\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-kube-api-access-nfv6h\") pod \"00c1f218-a462-4225-a443-69a3d29d5087\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " Jul 15 13:15:24.044937 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.044844 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-istio-token\") pod \"00c1f218-a462-4225-a443-69a3d29d5087\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " Jul 15 13:15:24.044937 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.044870 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-dns-cert\") pod \"00c1f218-a462-4225-a443-69a3d29d5087\" (UID: \"00c1f218-a462-4225-a443-69a3d29d5087\") " Jul 15 13:15:24.045187 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.045129 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "00c1f218-a462-4225-a443-69a3d29d5087" (UID: "00c1f218-a462-4225-a443-69a3d29d5087"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:15:24.047426 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.047318 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "00c1f218-a462-4225-a443-69a3d29d5087" (UID: "00c1f218-a462-4225-a443-69a3d29d5087"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:15:24.047426 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.047357 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-cacerts" (OuterVolumeSpecName: "cacerts") pod "00c1f218-a462-4225-a443-69a3d29d5087" (UID: "00c1f218-a462-4225-a443-69a3d29d5087"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:15:24.047871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.047808 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-istio-token" (OuterVolumeSpecName: "istio-token") pod "00c1f218-a462-4225-a443-69a3d29d5087" (UID: "00c1f218-a462-4225-a443-69a3d29d5087"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:15:24.048103 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.048079 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "00c1f218-a462-4225-a443-69a3d29d5087" (UID: "00c1f218-a462-4225-a443-69a3d29d5087"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:15:24.048190 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.048106 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-kube-api-access-nfv6h" (OuterVolumeSpecName: "kube-api-access-nfv6h") pod "00c1f218-a462-4225-a443-69a3d29d5087" (UID: "00c1f218-a462-4225-a443-69a3d29d5087"). InnerVolumeSpecName "kube-api-access-nfv6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:15:24.048455 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.048420 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00c1f218-a462-4225-a443-69a3d29d5087-local-certs" (OuterVolumeSpecName: "local-certs") pod "00c1f218-a462-4225-a443-69a3d29d5087" (UID: "00c1f218-a462-4225-a443-69a3d29d5087"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:15:24.146382 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.146349 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-ca-configmap\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:15:24.146382 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.146381 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-cacerts\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:15:24.146580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.146396 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00c1f218-a462-4225-a443-69a3d29d5087-local-certs\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:15:24.146580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.146409 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfv6h\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-kube-api-access-nfv6h\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:15:24.146580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.146420 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00c1f218-a462-4225-a443-69a3d29d5087-istio-token\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:15:24.146580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.146431 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-csr-dns-cert\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:15:24.146580 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.146443 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00c1f218-a462-4225-a443-69a3d29d5087-istio-kubeconfig\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:15:24.314643 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.314612 2582 generic.go:356] "Generic (PLEG): container finished" podID="00c1f218-a462-4225-a443-69a3d29d5087" containerID="367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a" exitCode=0 Jul 15 13:15:24.315176 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.314693 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" Jul 15 13:15:24.315176 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.314753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" event={"ID":"00c1f218-a462-4225-a443-69a3d29d5087","Type":"ContainerDied","Data":"367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a"} Jul 15 13:15:24.315176 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.314842 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn" event={"ID":"00c1f218-a462-4225-a443-69a3d29d5087","Type":"ContainerDied","Data":"a612f8b741380616dd5e1d2affd317206fa9b2d2e04fea5501e08e986f6ffc52"} Jul 15 13:15:24.315176 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.314869 2582 scope.go:117] "RemoveContainer" containerID="367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a" Jul 15 13:15:24.315680 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.315547 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:24.316589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.316536 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678v6d2s" Jul 15 13:15:24.329009 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:24.328985 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c1f218_a462_4225_a443_69a3d29d5087.slice\": RecentStats: unable to find data in memory cache]" Jul 15 13:15:24.329146 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:24.329117 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c1f218_a462_4225_a443_69a3d29d5087.slice\": RecentStats: unable to find data in memory cache]" Jul 15 13:15:24.357924 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.357879 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn"] Jul 15 13:15:24.361424 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.361397 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mgxpn"] Jul 15 13:15:24.435499 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.435468 2582 scope.go:117] "RemoveContainer" containerID="367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a" Jul 15 13:15:24.435948 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:15:24.435903 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a\": container with ID starting with 367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a not found: ID does not exist" containerID="367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a" Jul 15 13:15:24.436072 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:24.435959 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a"} err="failed to get container status \"367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a\": rpc error: code = NotFound desc = could not find container \"367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a\": container with ID starting with 367bedd7567eab5f971a1dea3a2fd27695bfba27595d52a345b5c79ab8964d8a not found: ID does not exist" Jul 15 13:15:25.462306 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:25.462276 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c1f218-a462-4225-a443-69a3d29d5087" path="/var/lib/kubelet/pods/00c1f218-a462-4225-a443-69a3d29d5087/volumes" Jul 15 13:15:26.267623 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:26.267577 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vs5qs" Jul 15 13:15:26.270089 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:26.270067 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-rp7bt" Jul 15 13:15:26.327826 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:26.327728 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" event={"ID":"786c6d59-4d31-48a3-a4e7-0c73d768d911","Type":"ContainerStarted","Data":"72be2235a7874472d4763152fa1dfa3ed0d3a0acda42ee919b08d603e6dd4c20"} Jul 15 13:15:26.328122 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:26.327894 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:26.345903 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:26.345846 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" podStartSLOduration=3.354424726 podStartE2EDuration="6.345826059s" totalImagesPullingTime="2.991401333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:26.344780868 +0000 UTC m=+541.514727222" watchObservedRunningTime="2026-07-15 13:15:26.345826059 +0000 UTC m=+541.515772414" Jul 15 13:15:27.333762 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:27.333683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" event={"ID":"216f8da5-74c0-425a-ba2c-e49228fc7295","Type":"ContainerStarted","Data":"42cae1174f059c989f438c1ae467ad0ed126e76778183f95ed8a6bd5b88ac6c6"} Jul 15 13:15:27.334301 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:27.333786 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:15:27.350448 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:27.350396 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" podStartSLOduration=3.274768818 podStartE2EDuration="7.350377952s" totalImagesPullingTime="4.075609134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:27.349694501 +0000 UTC m=+542.519640855" watchObservedRunningTime="2026-07-15 13:15:27.350377952 +0000 UTC m=+542.520324305" Jul 15 13:15:38.547103 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:38.547066 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:45.382364 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:45.382329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:15:57.339423 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:57.339396 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9bswr" Jul 15 13:15:58.340297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:15:58.340259 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lz7x" Jul 15 13:16:10.507845 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.507807 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:10.606381 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.606351 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rjr75"] Jul 15 13:16:10.606656 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.606643 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c1f218-a462-4225-a443-69a3d29d5087" containerName="discovery" Jul 15 13:16:10.606697 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.606658 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c1f218-a462-4225-a443-69a3d29d5087" containerName="discovery" Jul 15 13:16:10.606768 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.606740 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c1f218-a462-4225-a443-69a3d29d5087" containerName="discovery" Jul 15 13:16:10.614912 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.614894 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" Jul 15 13:16:10.617221 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.617193 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rjr75"] Jul 15 13:16:10.617850 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.617826 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 13:16:10.617850 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.617847 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 13:16:10.618019 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.617847 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-cds9h\"" Jul 15 13:16:10.618863 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.618841 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:10.648891 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.648869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlkk\" (UniqueName: \"kubernetes.io/projected/246de68d-acba-4a34-87c8-8ca04e8ae5fd-kube-api-access-ljlkk\") pod \"kuadrant-operator-catalog-rjr75\" (UID: \"246de68d-acba-4a34-87c8-8ca04e8ae5fd\") " pod="kuadrant-system/kuadrant-operator-catalog-rjr75" Jul 15 13:16:10.701441 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.701414 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:10.749298 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.749274 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlkk\" (UniqueName: \"kubernetes.io/projected/246de68d-acba-4a34-87c8-8ca04e8ae5fd-kube-api-access-ljlkk\") pod \"kuadrant-operator-catalog-rjr75\" (UID: \"246de68d-acba-4a34-87c8-8ca04e8ae5fd\") " pod="kuadrant-system/kuadrant-operator-catalog-rjr75" Jul 15 13:16:10.761559 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.761504 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlkk\" (UniqueName: \"kubernetes.io/projected/246de68d-acba-4a34-87c8-8ca04e8ae5fd-kube-api-access-ljlkk\") pod \"kuadrant-operator-catalog-rjr75\" (UID: \"246de68d-acba-4a34-87c8-8ca04e8ae5fd\") " pod="kuadrant-system/kuadrant-operator-catalog-rjr75" Jul 15 13:16:10.926814 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.926788 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" Jul 15 13:16:10.977189 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:10.977007 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rjr75"] Jul 15 13:16:11.053203 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.053014 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rjr75"] Jul 15 13:16:11.056028 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:16:11.056007 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246de68d_acba_4a34_87c8_8ca04e8ae5fd.slice/crio-c646e4f08112074e643108b816605e6f2f718c2691ff474916d70771e4a3996d WatchSource:0}: Error finding container c646e4f08112074e643108b816605e6f2f718c2691ff474916d70771e4a3996d: Status 404 returned error can't find the container with id c646e4f08112074e643108b816605e6f2f718c2691ff474916d70771e4a3996d Jul 15 13:16:11.187905 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.187880 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44zf4"] Jul 15 13:16:11.190548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.190531 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:11.199137 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.199116 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44zf4"] Jul 15 13:16:11.252253 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.252217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d6hl\" (UniqueName: \"kubernetes.io/projected/6d5f6fee-606e-4d74-855e-c55b0ec245ce-kube-api-access-5d6hl\") pod \"kuadrant-operator-catalog-44zf4\" (UID: \"6d5f6fee-606e-4d74-855e-c55b0ec245ce\") " pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:11.353534 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.353474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6hl\" (UniqueName: \"kubernetes.io/projected/6d5f6fee-606e-4d74-855e-c55b0ec245ce-kube-api-access-5d6hl\") pod \"kuadrant-operator-catalog-44zf4\" (UID: \"6d5f6fee-606e-4d74-855e-c55b0ec245ce\") " pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:11.361527 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.361467 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6hl\" (UniqueName: \"kubernetes.io/projected/6d5f6fee-606e-4d74-855e-c55b0ec245ce-kube-api-access-5d6hl\") pod \"kuadrant-operator-catalog-44zf4\" (UID: \"6d5f6fee-606e-4d74-855e-c55b0ec245ce\") " pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:11.425044 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.425004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:11.499998 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.499975 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:11.507919 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.507894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" event={"ID":"246de68d-acba-4a34-87c8-8ca04e8ae5fd","Type":"ContainerStarted","Data":"c646e4f08112074e643108b816605e6f2f718c2691ff474916d70771e4a3996d"} Jul 15 13:16:11.643846 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:11.641041 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-44zf4"] Jul 15 13:16:11.681563 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:16:11.681533 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5f6fee_606e_4d74_855e_c55b0ec245ce.slice/crio-b83b99a8b439703a613a0b4cae462495b4ee95f659c54a4cb61104fcca98fcee WatchSource:0}: Error finding container b83b99a8b439703a613a0b4cae462495b4ee95f659c54a4cb61104fcca98fcee: Status 404 returned error can't find the container with id b83b99a8b439703a613a0b4cae462495b4ee95f659c54a4cb61104fcca98fcee Jul 15 13:16:12.514002 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:12.513959 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" event={"ID":"6d5f6fee-606e-4d74-855e-c55b0ec245ce","Type":"ContainerStarted","Data":"b83b99a8b439703a613a0b4cae462495b4ee95f659c54a4cb61104fcca98fcee"} Jul 15 13:16:12.638708 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:12.638670 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:13.519109 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.519078 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" event={"ID":"246de68d-acba-4a34-87c8-8ca04e8ae5fd","Type":"ContainerStarted","Data":"229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b"} Jul 15 13:16:13.519517 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.519104 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" podUID="246de68d-acba-4a34-87c8-8ca04e8ae5fd" containerName="registry-server" containerID="cri-o://229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b" gracePeriod=2 Jul 15 13:16:13.520617 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.520596 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" event={"ID":"6d5f6fee-606e-4d74-855e-c55b0ec245ce","Type":"ContainerStarted","Data":"6c694658085b11dd1405bbb9322bb52175751c7c009d04b050b8ff7d8a42d44d"} Jul 15 13:16:13.533754 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.533694 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" podStartSLOduration=1.62545234 podStartE2EDuration="3.533678799s" totalImagesPullingTime="1.908226459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:16:13.53341725 +0000 UTC m=+588.703363621" watchObservedRunningTime="2026-07-15 13:16:13.533678799 +0000 UTC m=+588.703625156" Jul 15 13:16:13.548961 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.548924 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" podStartSLOduration=1.151047441 podStartE2EDuration="2.548912998s" totalImagesPullingTime="1.397865557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:16:13.548201869 +0000 UTC m=+588.718148223" watchObservedRunningTime="2026-07-15 13:16:13.548912998 +0000 UTC m=+588.718859361" Jul 15 13:16:13.756164 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.756141 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" Jul 15 13:16:13.774386 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.774331 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljlkk\" (UniqueName: \"kubernetes.io/projected/246de68d-acba-4a34-87c8-8ca04e8ae5fd-kube-api-access-ljlkk\") pod \"246de68d-acba-4a34-87c8-8ca04e8ae5fd\" (UID: \"246de68d-acba-4a34-87c8-8ca04e8ae5fd\") " Jul 15 13:16:13.776410 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.776373 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246de68d-acba-4a34-87c8-8ca04e8ae5fd-kube-api-access-ljlkk" (OuterVolumeSpecName: "kube-api-access-ljlkk") pod "246de68d-acba-4a34-87c8-8ca04e8ae5fd" (UID: "246de68d-acba-4a34-87c8-8ca04e8ae5fd"). InnerVolumeSpecName "kube-api-access-ljlkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:16:13.875767 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:13.875747 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlkk\" (UniqueName: \"kubernetes.io/projected/246de68d-acba-4a34-87c8-8ca04e8ae5fd-kube-api-access-ljlkk\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:16:14.525901 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.525871 2582 generic.go:356] "Generic (PLEG): container finished" podID="246de68d-acba-4a34-87c8-8ca04e8ae5fd" containerID="229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b" exitCode=0 Jul 15 13:16:14.526235 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.525912 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" Jul 15 13:16:14.526235 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.525954 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" event={"ID":"246de68d-acba-4a34-87c8-8ca04e8ae5fd","Type":"ContainerDied","Data":"229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b"} Jul 15 13:16:14.526235 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.525994 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rjr75" event={"ID":"246de68d-acba-4a34-87c8-8ca04e8ae5fd","Type":"ContainerDied","Data":"c646e4f08112074e643108b816605e6f2f718c2691ff474916d70771e4a3996d"} Jul 15 13:16:14.526235 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.526017 2582 scope.go:117] "RemoveContainer" containerID="229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b" Jul 15 13:16:14.535127 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.535110 2582 scope.go:117] "RemoveContainer" containerID="229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b" Jul 15 13:16:14.535376 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:16:14.535357 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b\": container with ID starting with 229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b not found: ID does not exist" containerID="229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b" Jul 15 13:16:14.535441 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.535382 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b"} err="failed to get container status \"229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b\": rpc error: code = NotFound desc = could not find container \"229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b\": container with ID starting with 229ffd09e4e424df1160f7eb93b75653d09786f7d8bd639dac47f578aa4cab5b not found: ID does not exist" Jul 15 13:16:14.544366 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.544343 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rjr75"] Jul 15 13:16:14.547848 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:14.547829 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rjr75"] Jul 15 13:16:15.459982 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:15.459945 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246de68d-acba-4a34-87c8-8ca04e8ae5fd" path="/var/lib/kubelet/pods/246de68d-acba-4a34-87c8-8ca04e8ae5fd/volumes" Jul 15 13:16:21.500637 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:21.500609 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:21.501024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:21.500683 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:21.522737 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:21.522703 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:21.575208 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:21.575187 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-44zf4" Jul 15 13:16:25.249842 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:25.249806 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:16:25.250244 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:25.250078 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:16:27.677366 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:27.677333 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:44.321700 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:44.321671 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:47.748936 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:47.748838 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:49.065000 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.064967 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd"] Jul 15 13:16:49.065357 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.065265 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246de68d-acba-4a34-87c8-8ca04e8ae5fd" containerName="registry-server" Jul 15 13:16:49.065357 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.065276 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="246de68d-acba-4a34-87c8-8ca04e8ae5fd" containerName="registry-server" Jul 15 13:16:49.065357 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.065330 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="246de68d-acba-4a34-87c8-8ca04e8ae5fd" containerName="registry-server" Jul 15 13:16:49.069294 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.069272 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:49.074547 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.074524 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fnrbn\"" Jul 15 13:16:49.085573 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.085553 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd"] Jul 15 13:16:49.122384 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.122358 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7f4l\" (UniqueName: \"kubernetes.io/projected/6ef2e95e-1810-4317-ae72-c2f3b5465226-kube-api-access-t7f4l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" (UID: \"6ef2e95e-1810-4317-ae72-c2f3b5465226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:49.122485 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.122412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ef2e95e-1810-4317-ae72-c2f3b5465226-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" (UID: \"6ef2e95e-1810-4317-ae72-c2f3b5465226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:49.223679 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.223656 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ef2e95e-1810-4317-ae72-c2f3b5465226-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" (UID: \"6ef2e95e-1810-4317-ae72-c2f3b5465226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:49.223769 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.223693 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f4l\" (UniqueName: \"kubernetes.io/projected/6ef2e95e-1810-4317-ae72-c2f3b5465226-kube-api-access-t7f4l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" (UID: \"6ef2e95e-1810-4317-ae72-c2f3b5465226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:49.223998 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.223982 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ef2e95e-1810-4317-ae72-c2f3b5465226-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" (UID: \"6ef2e95e-1810-4317-ae72-c2f3b5465226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:49.231621 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.231604 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7f4l\" (UniqueName: \"kubernetes.io/projected/6ef2e95e-1810-4317-ae72-c2f3b5465226-kube-api-access-t7f4l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" (UID: \"6ef2e95e-1810-4317-ae72-c2f3b5465226\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:49.378660 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.378613 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:49.458993 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.458674 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:49.513840 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.513810 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd"] Jul 15 13:16:49.516841 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:16:49.516817 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef2e95e_1810_4317_ae72_c2f3b5465226.slice/crio-b213d44c7bc86335adc7567d29f965247159ca4686a367045765fb6a1b278917 WatchSource:0}: Error finding container b213d44c7bc86335adc7567d29f965247159ca4686a367045765fb6a1b278917: Status 404 returned error can't find the container with id b213d44c7bc86335adc7567d29f965247159ca4686a367045765fb6a1b278917 Jul 15 13:16:49.519260 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.519239 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:16:49.613383 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.613351 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:49.664986 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.664921 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" event={"ID":"6ef2e95e-1810-4317-ae72-c2f3b5465226","Type":"ContainerStarted","Data":"b213d44c7bc86335adc7567d29f965247159ca4686a367045765fb6a1b278917"} Jul 15 13:16:49.716456 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.716431 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:49.816924 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:49.816901 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:50.211287 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:50.211249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:50.310416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:50.310382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:50.978155 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:50.978126 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp"] Jul 15 13:16:50.980497 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:50.980468 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:16:50.982563 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:50.982541 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dz4rk\"" Jul 15 13:16:50.991301 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:50.991280 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp"] Jul 15 13:16:51.036582 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:51.036556 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v8jz\" (UniqueName: \"kubernetes.io/projected/192920f9-84bc-4218-b6e8-83af9caa69e1-kube-api-access-4v8jz\") pod \"limitador-operator-controller-manager-859d7b78bd-shksp\" (UID: \"192920f9-84bc-4218-b6e8-83af9caa69e1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:16:51.137836 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:51.137806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v8jz\" (UniqueName: \"kubernetes.io/projected/192920f9-84bc-4218-b6e8-83af9caa69e1-kube-api-access-4v8jz\") pod \"limitador-operator-controller-manager-859d7b78bd-shksp\" (UID: \"192920f9-84bc-4218-b6e8-83af9caa69e1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:16:51.148518 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:51.148488 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v8jz\" (UniqueName: \"kubernetes.io/projected/192920f9-84bc-4218-b6e8-83af9caa69e1-kube-api-access-4v8jz\") pod \"limitador-operator-controller-manager-859d7b78bd-shksp\" (UID: \"192920f9-84bc-4218-b6e8-83af9caa69e1\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:16:51.291421 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:51.291360 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:16:51.315992 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:51.315964 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:51.440698 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:51.440672 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp"] Jul 15 13:16:51.444146 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:16:51.444121 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192920f9_84bc_4218_b6e8_83af9caa69e1.slice/crio-6ff6cf6184f7a5974d2f09f33e1794b900ff4b7f22edc78981526f4b84b43277 WatchSource:0}: Error finding container 6ff6cf6184f7a5974d2f09f33e1794b900ff4b7f22edc78981526f4b84b43277: Status 404 returned error can't find the container with id 6ff6cf6184f7a5974d2f09f33e1794b900ff4b7f22edc78981526f4b84b43277 Jul 15 13:16:51.676176 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:51.676106 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" event={"ID":"192920f9-84bc-4218-b6e8-83af9caa69e1","Type":"ContainerStarted","Data":"6ff6cf6184f7a5974d2f09f33e1794b900ff4b7f22edc78981526f4b84b43277"} Jul 15 13:16:51.763345 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:51.763310 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:53.142476 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:53.142440 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:53.686819 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:53.686738 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" event={"ID":"192920f9-84bc-4218-b6e8-83af9caa69e1","Type":"ContainerStarted","Data":"5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f"} Jul 15 13:16:53.686951 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:53.686842 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:16:53.703341 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:53.702697 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" podStartSLOduration=1.7322943130000001 podStartE2EDuration="3.702680151s" totalImagesPullingTime="1.970385838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:16:53.702134729 +0000 UTC m=+628.872081081" watchObservedRunningTime="2026-07-15 13:16:53.702680151 +0000 UTC m=+628.872626507" Jul 15 13:16:54.717914 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:54.717886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:55.536055 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:55.536027 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:56.701562 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:56.701533 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" event={"ID":"6ef2e95e-1810-4317-ae72-c2f3b5465226","Type":"ContainerStarted","Data":"a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd"} Jul 15 13:16:56.701948 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:56.701595 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:16:56.721656 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:56.721605 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" podStartSLOduration=0.915056812 podStartE2EDuration="7.721586452s" totalImagesPullingTime="6.80652964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:16:56.720033451 +0000 UTC m=+631.889979803" watchObservedRunningTime="2026-07-15 13:16:56.721586452 +0000 UTC m=+631.891532804" Jul 15 13:16:56.939707 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:56.939676 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w"] Jul 15 13:16:56.941975 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:56.941958 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" Jul 15 13:16:56.944512 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:56.944490 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 15 13:16:56.944512 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:56.944497 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g4pwb\"" Jul 15 13:16:56.956207 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:56.956162 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w"] Jul 15 13:16:57.084521 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:57.084489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhskr\" (UniqueName: \"kubernetes.io/projected/7c776e71-9420-44f4-a077-1969278300a1-kube-api-access-bhskr\") pod \"dns-operator-controller-manager-864dd78c4c-nth5w\" (UID: \"7c776e71-9420-44f4-a077-1969278300a1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" Jul 15 13:16:57.185979 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:57.185945 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhskr\" (UniqueName: \"kubernetes.io/projected/7c776e71-9420-44f4-a077-1969278300a1-kube-api-access-bhskr\") pod \"dns-operator-controller-manager-864dd78c4c-nth5w\" (UID: \"7c776e71-9420-44f4-a077-1969278300a1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" Jul 15 13:16:57.202397 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:57.202366 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhskr\" (UniqueName: \"kubernetes.io/projected/7c776e71-9420-44f4-a077-1969278300a1-kube-api-access-bhskr\") pod \"dns-operator-controller-manager-864dd78c4c-nth5w\" (UID: \"7c776e71-9420-44f4-a077-1969278300a1\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" Jul 15 13:16:57.252341 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:57.252268 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" Jul 15 13:16:57.384239 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:57.384207 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w"] Jul 15 13:16:57.387536 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:16:57.387511 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c776e71_9420_44f4_a077_1969278300a1.slice/crio-6c4d6ee766105fa1dbca9ba13d6460f344b026e82c67b54b5223fe5164df943b WatchSource:0}: Error finding container 6c4d6ee766105fa1dbca9ba13d6460f344b026e82c67b54b5223fe5164df943b: Status 404 returned error can't find the container with id 6c4d6ee766105fa1dbca9ba13d6460f344b026e82c67b54b5223fe5164df943b Jul 15 13:16:57.573737 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:57.573643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:57.706813 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:57.706778 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" event={"ID":"7c776e71-9420-44f4-a077-1969278300a1","Type":"ContainerStarted","Data":"6c4d6ee766105fa1dbca9ba13d6460f344b026e82c67b54b5223fe5164df943b"} Jul 15 13:16:59.263662 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.263630 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg"] Jul 15 13:16:59.266315 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.266291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.268248 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.268160 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 13:16:59.268415 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.268399 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 13:16:59.268493 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.268464 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2dxsb\"" Jul 15 13:16:59.275885 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.275861 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg"] Jul 15 13:16:59.283112 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.283087 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:59.380142 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.380120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:59.404634 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.404612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cb6207b-86f6-4769-a4e7-c274b6801a68-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.404795 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.404642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb6207b-86f6-4769-a4e7-c274b6801a68-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.404795 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.404661 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tv4\" (UniqueName: \"kubernetes.io/projected/0cb6207b-86f6-4769-a4e7-c274b6801a68-kube-api-access-z4tv4\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.479490 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.479461 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:16:59.505260 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.505236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cb6207b-86f6-4769-a4e7-c274b6801a68-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.505399 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.505267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb6207b-86f6-4769-a4e7-c274b6801a68-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.505399 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.505289 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tv4\" (UniqueName: \"kubernetes.io/projected/0cb6207b-86f6-4769-a4e7-c274b6801a68-kube-api-access-z4tv4\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.506055 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.506035 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cb6207b-86f6-4769-a4e7-c274b6801a68-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.507553 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.507533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cb6207b-86f6-4769-a4e7-c274b6801a68-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.512960 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.512935 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tv4\" (UniqueName: \"kubernetes.io/projected/0cb6207b-86f6-4769-a4e7-c274b6801a68-kube-api-access-z4tv4\") pod \"kuadrant-console-plugin-66b5774985-svgjg\" (UID: \"0cb6207b-86f6-4769-a4e7-c274b6801a68\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.577341 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.577285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" Jul 15 13:16:59.704504 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.704479 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg"] Jul 15 13:16:59.707078 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:16:59.707050 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cb6207b_86f6_4769_a4e7_c274b6801a68.slice/crio-d544a2fcf59242ad9ee66a6b5e60524344d44cedc90059079b77b22cded9eba1 WatchSource:0}: Error finding container d544a2fcf59242ad9ee66a6b5e60524344d44cedc90059079b77b22cded9eba1: Status 404 returned error can't find the container with id d544a2fcf59242ad9ee66a6b5e60524344d44cedc90059079b77b22cded9eba1 Jul 15 13:16:59.716779 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.716754 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" event={"ID":"7c776e71-9420-44f4-a077-1969278300a1","Type":"ContainerStarted","Data":"793bdefa6d448c3bb901f9e6ec10ef16b750573fb77d2dd70ebe974cf156cae6"} Jul 15 13:16:59.716894 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.716839 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" Jul 15 13:16:59.717922 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.717901 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" event={"ID":"0cb6207b-86f6-4769-a4e7-c274b6801a68","Type":"ContainerStarted","Data":"d544a2fcf59242ad9ee66a6b5e60524344d44cedc90059079b77b22cded9eba1"} Jul 15 13:16:59.738419 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:16:59.738378 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" podStartSLOduration=1.757858745 podStartE2EDuration="3.73836623s" totalImagesPullingTime="1.980507485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:16:59.737688087 +0000 UTC m=+634.907634464" watchObservedRunningTime="2026-07-15 13:16:59.73836623 +0000 UTC m=+634.908312583" Jul 15 13:17:00.347863 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:00.347834 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:00.416651 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:00.416621 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:04.693547 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:04.693513 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:06.921260 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:06.921222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:07.020048 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:07.020014 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:07.709141 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:07.709113 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.437025 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.436990 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd"] Jul 15 13:17:09.437400 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.437186 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" containerName="manager" containerID="cri-o://a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd" gracePeriod=2 Jul 15 13:17:09.462233 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.462210 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd"] Jul 15 13:17:09.471538 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.471500 2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"6ef2e95e-1810-4317-ae72-c2f3b5465226\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-66hfd\": pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.479218 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.479190 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk"] Jul 15 13:17:09.479651 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.479607 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" containerName="manager" Jul 15 13:17:09.479651 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.479629 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" containerName="manager" Jul 15 13:17:09.479840 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.479767 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" containerName="manager" Jul 15 13:17:09.481873 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.481854 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp"] Jul 15 13:17:09.481986 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.481953 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:09.482092 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.482020 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" containerName="manager" containerID="cri-o://5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f" gracePeriod=2 Jul 15 13:17:09.489906 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.489855 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp"] Jul 15 13:17:09.507223 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.507200 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk"] Jul 15 13:17:09.517907 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.517885 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr"] Jul 15 13:17:09.518232 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.518216 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" containerName="manager" Jul 15 13:17:09.518297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.518236 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" containerName="manager" Jul 15 13:17:09.518335 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.518299 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" containerName="manager" Jul 15 13:17:09.520309 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.520287 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" Jul 15 13:17:09.524017 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.523982 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.525860 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.525827 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:09.534036 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.533795 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr"] Jul 15 13:17:09.547063 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.547014 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.548936 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.548902 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:09.587477 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.587445 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgdrd\" (UniqueName: \"kubernetes.io/projected/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-kube-api-access-sgdrd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h2gdk\" (UID: \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:09.587610 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.587533 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h2gdk\" (UID: \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:09.688811 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.688755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h2gdk\" (UID: \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:09.688926 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.688816 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88skr\" (UniqueName: \"kubernetes.io/projected/e567bcad-3f98-45b3-90c7-33466e6c83c0-kube-api-access-88skr\") pod \"limitador-operator-controller-manager-859d7b78bd-tzzjr\" (UID: \"e567bcad-3f98-45b3-90c7-33466e6c83c0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" Jul 15 13:17:09.688926 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.688845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgdrd\" (UniqueName: \"kubernetes.io/projected/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-kube-api-access-sgdrd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h2gdk\" (UID: \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:09.689177 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.689153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h2gdk\" (UID: \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:09.696314 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.696296 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.697222 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.697198 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgdrd\" (UniqueName: \"kubernetes.io/projected/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-kube-api-access-sgdrd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h2gdk\" (UID: \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:09.698468 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.698448 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.699892 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.699872 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:09.717522 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.717504 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:09.719265 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.719235 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.720664 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.720642 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:09.770350 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.770327 2582 generic.go:356] "Generic (PLEG): container finished" podID="192920f9-84bc-4218-b6e8-83af9caa69e1" containerID="5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f" exitCode=0 Jul 15 13:17:09.770440 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.770361 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:09.770440 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.770428 2582 scope.go:117] "RemoveContainer" containerID="5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f" Jul 15 13:17:09.771665 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.771646 2582 generic.go:356] "Generic (PLEG): container finished" podID="6ef2e95e-1810-4317-ae72-c2f3b5465226" containerID="a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd" exitCode=0 Jul 15 13:17:09.771743 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.771698 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.772356 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.772333 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.773848 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.773822 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:09.775251 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.775229 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:09.776642 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.776617 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:09.778368 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.778351 2582 scope.go:117] "RemoveContainer" containerID="5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f" Jul 15 13:17:09.778605 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.778585 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f\": container with ID starting with 5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f not found: ID does not exist" containerID="5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f" Jul 15 13:17:09.778674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.778613 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f"} err="failed to get container status \"5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f\": rpc error: code = NotFound desc = could not find container \"5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f\": container with ID starting with 5c8bb9444e71f9531b29f36397c2c035ce511ab750a4e7dd43bdbb03e2b55f6f not found: ID does not exist" Jul 15 13:17:09.778674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.778629 2582 scope.go:117] "RemoveContainer" containerID="a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd" Jul 15 13:17:09.785603 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.785586 2582 scope.go:117] "RemoveContainer" containerID="a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd" Jul 15 13:17:09.785861 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:09.785845 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd\": container with ID starting with a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd not found: ID does not exist" containerID="a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd" Jul 15 13:17:09.785914 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.785875 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd"} err="failed to get container status \"a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd\": rpc error: code = NotFound desc = could not find container \"a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd\": container with ID starting with a29a3148f6b863aba55a8eec6ef3371eb3949d362a5b98d11696c5b0946c8afd not found: ID does not exist" Jul 15 13:17:09.789069 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.789051 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v8jz\" (UniqueName: \"kubernetes.io/projected/192920f9-84bc-4218-b6e8-83af9caa69e1-kube-api-access-4v8jz\") pod \"192920f9-84bc-4218-b6e8-83af9caa69e1\" (UID: \"192920f9-84bc-4218-b6e8-83af9caa69e1\") " Jul 15 13:17:09.789129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.789092 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ef2e95e-1810-4317-ae72-c2f3b5465226-extensions-socket-volume\") pod \"6ef2e95e-1810-4317-ae72-c2f3b5465226\" (UID: \"6ef2e95e-1810-4317-ae72-c2f3b5465226\") " Jul 15 13:17:09.789162 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.789150 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7f4l\" (UniqueName: \"kubernetes.io/projected/6ef2e95e-1810-4317-ae72-c2f3b5465226-kube-api-access-t7f4l\") pod \"6ef2e95e-1810-4317-ae72-c2f3b5465226\" (UID: \"6ef2e95e-1810-4317-ae72-c2f3b5465226\") " Jul 15 13:17:09.789227 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.789214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88skr\" (UniqueName: \"kubernetes.io/projected/e567bcad-3f98-45b3-90c7-33466e6c83c0-kube-api-access-88skr\") pod \"limitador-operator-controller-manager-859d7b78bd-tzzjr\" (UID: \"e567bcad-3f98-45b3-90c7-33466e6c83c0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" Jul 15 13:17:09.789555 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.789532 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef2e95e-1810-4317-ae72-c2f3b5465226-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6ef2e95e-1810-4317-ae72-c2f3b5465226" (UID: "6ef2e95e-1810-4317-ae72-c2f3b5465226"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:17:09.790907 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.790887 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192920f9-84bc-4218-b6e8-83af9caa69e1-kube-api-access-4v8jz" (OuterVolumeSpecName: "kube-api-access-4v8jz") pod "192920f9-84bc-4218-b6e8-83af9caa69e1" (UID: "192920f9-84bc-4218-b6e8-83af9caa69e1"). InnerVolumeSpecName "kube-api-access-4v8jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:17:09.790993 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.790975 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef2e95e-1810-4317-ae72-c2f3b5465226-kube-api-access-t7f4l" (OuterVolumeSpecName: "kube-api-access-t7f4l") pod "6ef2e95e-1810-4317-ae72-c2f3b5465226" (UID: "6ef2e95e-1810-4317-ae72-c2f3b5465226"). InnerVolumeSpecName "kube-api-access-t7f4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:17:09.800991 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.800969 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88skr\" (UniqueName: \"kubernetes.io/projected/e567bcad-3f98-45b3-90c7-33466e6c83c0-kube-api-access-88skr\") pod \"limitador-operator-controller-manager-859d7b78bd-tzzjr\" (UID: \"e567bcad-3f98-45b3-90c7-33466e6c83c0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" Jul 15 13:17:09.852079 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.852054 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:09.859964 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.859942 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" Jul 15 13:17:09.890410 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.890376 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ef2e95e-1810-4317-ae72-c2f3b5465226-extensions-socket-volume\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:17:09.890410 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.890411 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7f4l\" (UniqueName: \"kubernetes.io/projected/6ef2e95e-1810-4317-ae72-c2f3b5465226-kube-api-access-t7f4l\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:17:09.890609 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:09.890427 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v8jz\" (UniqueName: \"kubernetes.io/projected/192920f9-84bc-4218-b6e8-83af9caa69e1-kube-api-access-4v8jz\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:17:10.013332 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.013309 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk"] Jul 15 13:17:10.015429 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:17:10.015403 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfffa88_0636_4a2a_aa29_50c0ce97db5f.slice/crio-ba153b49cfac81df6dd28375223d83fe69001c71ee5446905b7224a1ce439d72 WatchSource:0}: Error finding container ba153b49cfac81df6dd28375223d83fe69001c71ee5446905b7224a1ce439d72: Status 404 returned error can't find the container with id ba153b49cfac81df6dd28375223d83fe69001c71ee5446905b7224a1ce439d72 Jul 15 13:17:10.037374 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.037348 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr"] Jul 15 13:17:10.046866 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:17:10.046840 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode567bcad_3f98_45b3_90c7_33466e6c83c0.slice/crio-65b82a40d0ea51471b0102bc4726c8dcf71e3e189c41b5fbdb21bd2fb4d3d151 WatchSource:0}: Error finding container 65b82a40d0ea51471b0102bc4726c8dcf71e3e189c41b5fbdb21bd2fb4d3d151: Status 404 returned error can't find the container with id 65b82a40d0ea51471b0102bc4726c8dcf71e3e189c41b5fbdb21bd2fb4d3d151 Jul 15 13:17:10.085630 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.085601 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:10.087659 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.087629 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:10.089199 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.089175 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:10.090599 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.090574 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:10.725215 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.725137 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nth5w" Jul 15 13:17:10.727204 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.727173 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:10.728886 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.728843 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:10.779166 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.779134 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" event={"ID":"e567bcad-3f98-45b3-90c7-33466e6c83c0","Type":"ContainerStarted","Data":"00368e786b0f348bbe19e4391badcca68503d5499e83057c78d89cf5364847f7"} Jul 15 13:17:10.779335 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.779187 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" event={"ID":"e567bcad-3f98-45b3-90c7-33466e6c83c0","Type":"ContainerStarted","Data":"65b82a40d0ea51471b0102bc4726c8dcf71e3e189c41b5fbdb21bd2fb4d3d151"} Jul 15 13:17:10.779402 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.779351 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" Jul 15 13:17:10.783491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.783462 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" event={"ID":"ccfffa88-0636-4a2a-aa29-50c0ce97db5f","Type":"ContainerStarted","Data":"96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751"} Jul 15 13:17:10.783604 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.783502 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" event={"ID":"ccfffa88-0636-4a2a-aa29-50c0ce97db5f","Type":"ContainerStarted","Data":"ba153b49cfac81df6dd28375223d83fe69001c71ee5446905b7224a1ce439d72"} Jul 15 13:17:10.783604 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.783559 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:10.801796 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.801762 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:10.802121 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.802070 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" podStartSLOduration=1.802054156 podStartE2EDuration="1.802054156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:17:10.800278064 +0000 UTC m=+645.970224437" watchObservedRunningTime="2026-07-15 13:17:10.802054156 +0000 UTC m=+645.972000510" Jul 15 13:17:10.803406 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.803339 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:10.804961 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.804934 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-66hfd\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-66hfd" Jul 15 13:17:10.806651 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:10.806623 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-shksp\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-shksp" Jul 15 13:17:10.829444 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:10.829402 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" podStartSLOduration=1.8293844369999999 podStartE2EDuration="1.829384437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:17:10.827349302 +0000 UTC m=+645.997295652" watchObservedRunningTime="2026-07-15 13:17:10.829384437 +0000 UTC m=+645.999330786" Jul 15 13:17:11.462369 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:11.462320 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192920f9-84bc-4218-b6e8-83af9caa69e1" path="/var/lib/kubelet/pods/192920f9-84bc-4218-b6e8-83af9caa69e1/volumes" Jul 15 13:17:11.462903 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:11.462880 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef2e95e-1810-4317-ae72-c2f3b5465226" path="/var/lib/kubelet/pods/6ef2e95e-1810-4317-ae72-c2f3b5465226/volumes" Jul 15 13:17:11.746598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:11.746528 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:11.848559 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:11.848526 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:14.184524 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:14.184495 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:15.023502 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:15.023467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:19.720427 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:19.720361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:19.820224 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:19.820189 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:21.790233 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:21.790198 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:21.790661 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:21.790291 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tzzjr" Jul 15 13:17:24.857211 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:24.857176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" event={"ID":"0cb6207b-86f6-4769-a4e7-c274b6801a68","Type":"ContainerStarted","Data":"e2b47ba5e3f16d53cd02031c2c3b514c19e5b0218fb760cfe850f60649ee3b5b"} Jul 15 13:17:24.873432 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:24.873387 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-svgjg" podStartSLOduration=1.13614618 podStartE2EDuration="25.873371039s" totalImagesPullingTime="24.737224859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:17:24.872668556 +0000 UTC m=+660.042614909" watchObservedRunningTime="2026-07-15 13:17:24.873371039 +0000 UTC m=+660.043317392" Jul 15 13:17:26.623079 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:26.623004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:26.726623 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:26.726591 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:28.206036 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.206006 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk"] Jul 15 13:17:28.206556 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.206246 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" podUID="ccfffa88-0636-4a2a-aa29-50c0ce97db5f" containerName="manager" containerID="cri-o://96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751" gracePeriod=10 Jul 15 13:17:28.394891 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.394857 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn"] Jul 15 13:17:28.417830 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.417794 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn"] Jul 15 13:17:28.418017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.417915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:28.443663 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.443635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30e69da1-fce2-4054-93fe-d0c042e421bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sfzhn\" (UID: \"30e69da1-fce2-4054-93fe-d0c042e421bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:28.443830 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.443684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74cj\" (UniqueName: \"kubernetes.io/projected/30e69da1-fce2-4054-93fe-d0c042e421bc-kube-api-access-s74cj\") pod \"kuadrant-operator-controller-manager-86878c769f-sfzhn\" (UID: \"30e69da1-fce2-4054-93fe-d0c042e421bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:28.497892 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.497867 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:28.544214 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.544190 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgdrd\" (UniqueName: \"kubernetes.io/projected/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-kube-api-access-sgdrd\") pod \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\" (UID: \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\") " Jul 15 13:17:28.544366 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.544245 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-extensions-socket-volume\") pod \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\" (UID: \"ccfffa88-0636-4a2a-aa29-50c0ce97db5f\") " Jul 15 13:17:28.544366 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.544339 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30e69da1-fce2-4054-93fe-d0c042e421bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sfzhn\" (UID: \"30e69da1-fce2-4054-93fe-d0c042e421bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:28.544479 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.544395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74cj\" (UniqueName: \"kubernetes.io/projected/30e69da1-fce2-4054-93fe-d0c042e421bc-kube-api-access-s74cj\") pod \"kuadrant-operator-controller-manager-86878c769f-sfzhn\" (UID: \"30e69da1-fce2-4054-93fe-d0c042e421bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:28.544650 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.544626 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ccfffa88-0636-4a2a-aa29-50c0ce97db5f" (UID: "ccfffa88-0636-4a2a-aa29-50c0ce97db5f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:17:28.544873 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.544851 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30e69da1-fce2-4054-93fe-d0c042e421bc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sfzhn\" (UID: \"30e69da1-fce2-4054-93fe-d0c042e421bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:28.546372 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.546350 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-kube-api-access-sgdrd" (OuterVolumeSpecName: "kube-api-access-sgdrd") pod "ccfffa88-0636-4a2a-aa29-50c0ce97db5f" (UID: "ccfffa88-0636-4a2a-aa29-50c0ce97db5f"). InnerVolumeSpecName "kube-api-access-sgdrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:17:28.553391 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.553372 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74cj\" (UniqueName: \"kubernetes.io/projected/30e69da1-fce2-4054-93fe-d0c042e421bc-kube-api-access-s74cj\") pod \"kuadrant-operator-controller-manager-86878c769f-sfzhn\" (UID: \"30e69da1-fce2-4054-93fe-d0c042e421bc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:28.645623 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.645597 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-extensions-socket-volume\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:17:28.645623 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.645620 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgdrd\" (UniqueName: \"kubernetes.io/projected/ccfffa88-0636-4a2a-aa29-50c0ce97db5f-kube-api-access-sgdrd\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:17:28.728416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.728347 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:28.860104 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.860066 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn"] Jul 15 13:17:28.862733 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:17:28.862687 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e69da1_fce2_4054_93fe_d0c042e421bc.slice/crio-3bf88af12bb1b67d378ab1b076252fcefef98444f89d7a76eae91679a67ebe6c WatchSource:0}: Error finding container 3bf88af12bb1b67d378ab1b076252fcefef98444f89d7a76eae91679a67ebe6c: Status 404 returned error can't find the container with id 3bf88af12bb1b67d378ab1b076252fcefef98444f89d7a76eae91679a67ebe6c Jul 15 13:17:28.879872 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.879835 2582 generic.go:356] "Generic (PLEG): container finished" podID="ccfffa88-0636-4a2a-aa29-50c0ce97db5f" containerID="96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751" exitCode=0 Jul 15 13:17:28.879928 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.879891 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" Jul 15 13:17:28.879963 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.879919 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" event={"ID":"ccfffa88-0636-4a2a-aa29-50c0ce97db5f","Type":"ContainerDied","Data":"96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751"} Jul 15 13:17:28.880020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.879995 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk" event={"ID":"ccfffa88-0636-4a2a-aa29-50c0ce97db5f","Type":"ContainerDied","Data":"ba153b49cfac81df6dd28375223d83fe69001c71ee5446905b7224a1ce439d72"} Jul 15 13:17:28.880061 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.880039 2582 scope.go:117] "RemoveContainer" containerID="96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751" Jul 15 13:17:28.881332 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.881309 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" event={"ID":"30e69da1-fce2-4054-93fe-d0c042e421bc","Type":"ContainerStarted","Data":"3bf88af12bb1b67d378ab1b076252fcefef98444f89d7a76eae91679a67ebe6c"} Jul 15 13:17:28.911668 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.911646 2582 scope.go:117] "RemoveContainer" containerID="96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751" Jul 15 13:17:28.911959 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:17:28.911939 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751\": container with ID starting with 96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751 not found: ID does not exist" containerID="96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751" Jul 15 13:17:28.912014 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.911967 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751"} err="failed to get container status \"96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751\": rpc error: code = NotFound desc = could not find container \"96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751\": container with ID starting with 96eb8b17e5d41e38b4310c31fbe624943ddb0f9be4adee69b4d4828926b8c751 not found: ID does not exist" Jul 15 13:17:28.922306 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.922282 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk"] Jul 15 13:17:28.928460 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:28.928442 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h2gdk"] Jul 15 13:17:29.460451 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:29.460419 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccfffa88-0636-4a2a-aa29-50c0ce97db5f" path="/var/lib/kubelet/pods/ccfffa88-0636-4a2a-aa29-50c0ce97db5f/volumes" Jul 15 13:17:29.887595 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:29.887511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" event={"ID":"30e69da1-fce2-4054-93fe-d0c042e421bc","Type":"ContainerStarted","Data":"574a0c19615836d2243b4f796c84a99fdebd7f3e5b9f447a1a0aac0f06fc83ef"} Jul 15 13:17:29.887786 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:29.887674 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:29.908393 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:29.908350 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" podStartSLOduration=1.908338249 podStartE2EDuration="1.908338249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:17:29.907093675 +0000 UTC m=+665.077040027" watchObservedRunningTime="2026-07-15 13:17:29.908338249 +0000 UTC m=+665.078284661" Jul 15 13:17:31.421954 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:31.421915 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:31.522667 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:31.522641 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:33.908341 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:33.908305 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:34.008754 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:34.008708 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:39.649668 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:39.649633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:39.716210 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:39.716183 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:40.894679 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:40.894647 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:17:44.562387 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.562352 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm"] Jul 15 13:17:44.562865 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.562650 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccfffa88-0636-4a2a-aa29-50c0ce97db5f" containerName="manager" Jul 15 13:17:44.562865 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.562661 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccfffa88-0636-4a2a-aa29-50c0ce97db5f" containerName="manager" Jul 15 13:17:44.562865 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.562742 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccfffa88-0636-4a2a-aa29-50c0ce97db5f" containerName="manager" Jul 15 13:17:44.601987 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.601960 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:44.606774 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.606749 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm"] Jul 15 13:17:44.606889 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.606836 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.608956 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.608935 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kzlbj\"" Jul 15 13:17:44.663118 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.663287 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.663287 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663150 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.663287 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcjb\" (UniqueName: \"kubernetes.io/projected/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-kube-api-access-kvcjb\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.663466 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663310 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.663466 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663337 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.663466 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663447 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.663616 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663485 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.663616 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.663552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.747645 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.747615 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:44.764783 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.764760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.764870 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.764787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.764870 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.764815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.764870 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.764846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.764871 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.764910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.764940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcjb\" (UniqueName: \"kubernetes.io/projected/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-kube-api-access-kvcjb\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.764982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.765008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765308 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.765286 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765529 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.765503 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765661 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.765554 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765661 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.765602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.765784 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.765745 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.767177 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.767157 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.767496 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.767478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.775345 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.775317 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.775345 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.775333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcjb\" (UniqueName: \"kubernetes.io/projected/5189b6ce-d057-47a4-80bb-6f89a2eb4b69-kube-api-access-kvcjb\") pod \"maas-default-gateway-openshift-default-5896d4dd49-2mlzm\" (UID: \"5189b6ce-d057-47a4-80bb-6f89a2eb4b69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:44.917910 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:44.917834 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:45.071782 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:45.071705 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm"] Jul 15 13:17:45.073927 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:17:45.073899 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5189b6ce_d057_47a4_80bb_6f89a2eb4b69.slice/crio-2084647e0fe0dc310b4652f88a116e266789d486c5700450d7f38a62842965ee WatchSource:0}: Error finding container 2084647e0fe0dc310b4652f88a116e266789d486c5700450d7f38a62842965ee: Status 404 returned error can't find the container with id 2084647e0fe0dc310b4652f88a116e266789d486c5700450d7f38a62842965ee Jul 15 13:17:45.076011 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:45.075982 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:17:45.076094 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:45.076045 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:17:45.076094 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:45.076073 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:17:45.964319 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:45.964280 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" event={"ID":"5189b6ce-d057-47a4-80bb-6f89a2eb4b69","Type":"ContainerStarted","Data":"72ea8810ac37b4848bfd4e2c838097daa0bdb900070c939c8cad237ef3a48230"} Jul 15 13:17:45.964686 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:45.964377 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" event={"ID":"5189b6ce-d057-47a4-80bb-6f89a2eb4b69","Type":"ContainerStarted","Data":"2084647e0fe0dc310b4652f88a116e266789d486c5700450d7f38a62842965ee"} Jul 15 13:17:45.984641 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:45.984586 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" podStartSLOduration=1.984568129 podStartE2EDuration="1.984568129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:17:45.984075684 +0000 UTC m=+681.154022038" watchObservedRunningTime="2026-07-15 13:17:45.984568129 +0000 UTC m=+681.154514483" Jul 15 13:17:46.918024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:46.917997 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:46.922522 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:46.922501 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:46.968440 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:46.968416 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:46.969093 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:46.969079 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:17:49.631802 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:49.631770 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:49.735333 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:49.735299 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:53.849813 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:53.849765 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:53.953161 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:53.953128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:56.298556 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:56.298518 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:17:56.398890 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:17:56.398860 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:00.026788 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.026756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:00.124595 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.124562 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:00.224193 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.224163 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:00.341855 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.341781 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:18:00.377396 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.377360 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:18:00.377598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.377451 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:00.379837 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.379812 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 13:18:00.380297 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.380279 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:18:00.481676 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.481652 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6086fdef-2573-413b-98c9-2f0b0e06d53e-config-file\") pod \"limitador-limitador-54d6d988d5-8rqs9\" (UID: \"6086fdef-2573-413b-98c9-2f0b0e06d53e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:00.481834 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.481741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xspzt\" (UniqueName: \"kubernetes.io/projected/6086fdef-2573-413b-98c9-2f0b0e06d53e-kube-api-access-xspzt\") pod \"limitador-limitador-54d6d988d5-8rqs9\" (UID: \"6086fdef-2573-413b-98c9-2f0b0e06d53e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:00.582706 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.582674 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xspzt\" (UniqueName: \"kubernetes.io/projected/6086fdef-2573-413b-98c9-2f0b0e06d53e-kube-api-access-xspzt\") pod \"limitador-limitador-54d6d988d5-8rqs9\" (UID: \"6086fdef-2573-413b-98c9-2f0b0e06d53e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:00.582860 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.582734 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6086fdef-2573-413b-98c9-2f0b0e06d53e-config-file\") pod \"limitador-limitador-54d6d988d5-8rqs9\" (UID: \"6086fdef-2573-413b-98c9-2f0b0e06d53e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:00.583305 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.583289 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6086fdef-2573-413b-98c9-2f0b0e06d53e-config-file\") pod \"limitador-limitador-54d6d988d5-8rqs9\" (UID: \"6086fdef-2573-413b-98c9-2f0b0e06d53e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:00.591687 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.591666 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xspzt\" (UniqueName: \"kubernetes.io/projected/6086fdef-2573-413b-98c9-2f0b0e06d53e-kube-api-access-xspzt\") pod \"limitador-limitador-54d6d988d5-8rqs9\" (UID: \"6086fdef-2573-413b-98c9-2f0b0e06d53e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:00.688917 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.688860 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:00.829242 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.829212 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:18:00.833178 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:18:00.833152 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6086fdef_2573_413b_98c9_2f0b0e06d53e.slice/crio-2683fecabc752cc7ca8e91ffeb64e3753ce9e5093b46d634fd27529645a6c07a WatchSource:0}: Error finding container 2683fecabc752cc7ca8e91ffeb64e3753ce9e5093b46d634fd27529645a6c07a: Status 404 returned error can't find the container with id 2683fecabc752cc7ca8e91ffeb64e3753ce9e5093b46d634fd27529645a6c07a Jul 15 13:18:00.870193 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.870167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:00.968881 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:00.968852 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:01.026756 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:01.026726 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" event={"ID":"6086fdef-2573-413b-98c9-2f0b0e06d53e","Type":"ContainerStarted","Data":"2683fecabc752cc7ca8e91ffeb64e3753ce9e5093b46d634fd27529645a6c07a"} Jul 15 13:18:04.043174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:04.043138 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" event={"ID":"6086fdef-2573-413b-98c9-2f0b0e06d53e","Type":"ContainerStarted","Data":"d8708af90df710f5d319f0d384b825fa5cca4e068e0ca7d37d7490768fe94a2e"} Jul 15 13:18:04.043642 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:04.043247 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:04.061915 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:04.061870 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" podStartSLOduration=1.528060338 podStartE2EDuration="4.061855686s" totalImagesPullingTime="2.533795348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:18:04.060495997 +0000 UTC m=+699.230442350" watchObservedRunningTime="2026-07-15 13:18:04.061855686 +0000 UTC m=+699.231802075" Jul 15 13:18:15.051434 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:15.051404 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8rqs9" Jul 15 13:18:18.472586 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:18.472512 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:19.340618 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:19.340577 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:19.447012 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:19.446980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:30.868555 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:30.868521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:31.242612 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:31.242568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:37.376286 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:37.376251 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:37.482077 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:37.482045 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:37.555814 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:37.555789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:40.464115 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:40.464081 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:18:40.636954 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:18:40.636914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:13.332173 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:13.332141 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:14.088894 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:14.088833 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:14.187977 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:14.187946 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:14.457022 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:14.456991 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:14.547811 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:14.547783 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:14.614372 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:14.614344 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:19:22.394985 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:22.394916 2582 generic.go:356] "Generic (PLEG): container finished" podID="5189b6ce-d057-47a4-80bb-6f89a2eb4b69" containerID="72ea8810ac37b4848bfd4e2c838097daa0bdb900070c939c8cad237ef3a48230" exitCode=0 Jul 15 13:19:22.395341 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:22.394980 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" event={"ID":"5189b6ce-d057-47a4-80bb-6f89a2eb4b69","Type":"ContainerDied","Data":"72ea8810ac37b4848bfd4e2c838097daa0bdb900070c939c8cad237ef3a48230"} Jul 15 13:19:22.395341 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:22.395333 2582 scope.go:117] "RemoveContainer" containerID="72ea8810ac37b4848bfd4e2c838097daa0bdb900070c939c8cad237ef3a48230" Jul 15 13:19:22.396352 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:22.396324 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:19:22.396422 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:22.396404 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:19:22.396471 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:22.396456 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:19:23.401231 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:23.401195 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" event={"ID":"5189b6ce-d057-47a4-80bb-6f89a2eb4b69","Type":"ContainerStarted","Data":"e3b47f4138d6085f76bd300536a12e56e510fdff84d6425584e8782707223d85"} Jul 15 13:19:23.918529 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:23.918494 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:19:23.918691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:23.918670 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-2mlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 15 13:19:23.918757 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:23.918740 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" podUID="5189b6ce-d057-47a4-80bb-6f89a2eb4b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 15 13:19:24.815131 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:24.815095 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:24.916583 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:24.916549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:24.918700 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:24.918676 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-2mlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 15 13:19:24.918851 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:24.918747 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" podUID="5189b6ce-d057-47a4-80bb-6f89a2eb4b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 15 13:19:25.918410 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:25.918374 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-2mlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 15 13:19:25.918894 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:25.918447 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" podUID="5189b6ce-d057-47a4-80bb-6f89a2eb4b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 15 13:19:26.918538 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:26.918503 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-2mlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Jul 15 13:19:26.918934 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:26.918556 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" podUID="5189b6ce-d057-47a4-80bb-6f89a2eb4b69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Jul 15 13:19:27.922636 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:27.922607 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:19:27.923001 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:27.922944 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:19:27.923687 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:27.923669 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-2mlzm" Jul 15 13:19:35.702816 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:35.702780 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:36.363855 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:36.363821 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:19:38.638338 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:38.638307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:39.434783 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:39.434752 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:19:52.150133 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:52.150064 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:19:53.306951 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:19:53.306909 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:20:01.018448 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:01.018415 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:20:01.125910 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:01.125877 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:20:04.307943 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:04.307910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:20:05.803606 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:05.803553 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:20:14.545447 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.545410 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc"] Jul 15 13:20:14.548834 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.548816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.551400 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.551363 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 13:20:14.551517 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.551408 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 13:20:14.552082 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.552063 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6wk5z\"" Jul 15 13:20:14.552194 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.552133 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 13:20:14.567386 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.567358 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc"] Jul 15 13:20:14.575179 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.575161 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:20:14.607776 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.607755 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.607868 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.607783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.607868 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.607806 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6x8f\" (UniqueName: \"kubernetes.io/projected/83874060-a0a8-4749-851f-dcd8262ce70d-kube-api-access-n6x8f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.607868 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.607838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83874060-a0a8-4749-851f-dcd8262ce70d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.607981 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.607906 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.607981 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.607945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.708828 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.708796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.708997 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.708840 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.708997 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.708976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.709121 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.709026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.709121 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.709065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6x8f\" (UniqueName: \"kubernetes.io/projected/83874060-a0a8-4749-851f-dcd8262ce70d-kube-api-access-n6x8f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.709218 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.709130 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83874060-a0a8-4749-851f-dcd8262ce70d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.709218 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.709176 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.709320 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.709224 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.709320 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.709258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.711053 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.711033 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83874060-a0a8-4749-851f-dcd8262ce70d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.711337 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.711319 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83874060-a0a8-4749-851f-dcd8262ce70d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.716806 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.716776 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6x8f\" (UniqueName: \"kubernetes.io/projected/83874060-a0a8-4749-851f-dcd8262ce70d-kube-api-access-n6x8f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2g7fc\" (UID: \"83874060-a0a8-4749-851f-dcd8262ce70d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.859111 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.859052 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:14.992798 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:14.990249 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc"] Jul 15 13:20:14.994157 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:20:14.994123 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83874060_a0a8_4749_851f_dcd8262ce70d.slice/crio-d9deb2d54d0311036afb07e81b4d93b5f89c0c3383f3ee99bb936ef09ee52879 WatchSource:0}: Error finding container d9deb2d54d0311036afb07e81b4d93b5f89c0c3383f3ee99bb936ef09ee52879: Status 404 returned error can't find the container with id d9deb2d54d0311036afb07e81b4d93b5f89c0c3383f3ee99bb936ef09ee52879 Jul 15 13:20:15.643970 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:15.643927 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" event={"ID":"83874060-a0a8-4749-851f-dcd8262ce70d","Type":"ContainerStarted","Data":"d9deb2d54d0311036afb07e81b4d93b5f89c0c3383f3ee99bb936ef09ee52879"} Jul 15 13:20:16.586700 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:16.586654 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:20:18.659849 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:18.659816 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" event={"ID":"83874060-a0a8-4749-851f-dcd8262ce70d","Type":"ContainerStarted","Data":"07e7aa157e0d3468413cb8251adf3641f33050a930f1f2b0710531c2026c3fed"} Jul 15 13:20:18.676667 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:18.676607 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" podStartSLOduration=1.88608501 podStartE2EDuration="4.676588138s" totalImagesPullingTime="2.790503128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:20:18.676374946 +0000 UTC m=+833.846321299" watchObservedRunningTime="2026-07-15 13:20:18.676588138 +0000 UTC m=+833.846534491" Jul 15 13:20:24.859646 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:24.859609 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:24.860092 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:24.859665 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:24.872256 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:24.872233 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:25.701487 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:25.701458 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2g7fc" Jul 15 13:20:27.679419 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.679387 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5"] Jul 15 13:20:27.683118 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.683098 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.685303 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.685283 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 13:20:27.694860 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.694841 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5"] Jul 15 13:20:27.747010 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.746975 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:20:27.811955 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.811924 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.811955 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.811961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.812180 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.812001 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.812180 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.812048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.812180 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.812070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.812180 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.812092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2bh\" (UniqueName: \"kubernetes.io/projected/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-kube-api-access-nl2bh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913274 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913274 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913519 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913519 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2bh\" (UniqueName: \"kubernetes.io/projected/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-kube-api-access-nl2bh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913519 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913519 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913752 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913624 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913824 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913779 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.913867 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.913835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.915585 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.915559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.915794 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.915777 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.921804 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.921784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2bh\" (UniqueName: \"kubernetes.io/projected/eb61a06f-a9d4-4f75-a4c8-64d5b208578f-kube-api-access-nl2bh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5\" (UID: \"eb61a06f-a9d4-4f75-a4c8-64d5b208578f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:27.992187 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:27.992152 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:28.128534 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:28.128508 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5"] Jul 15 13:20:28.131024 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:20:28.130990 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb61a06f_a9d4_4f75_a4c8_64d5b208578f.slice/crio-4e98c0151da94e653d421d37a2d03da09450ddb86ee37d6ed42781ac209d7040 WatchSource:0}: Error finding container 4e98c0151da94e653d421d37a2d03da09450ddb86ee37d6ed42781ac209d7040: Status 404 returned error can't find the container with id 4e98c0151da94e653d421d37a2d03da09450ddb86ee37d6ed42781ac209d7040 Jul 15 13:20:28.704975 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:28.704939 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" event={"ID":"eb61a06f-a9d4-4f75-a4c8-64d5b208578f","Type":"ContainerStarted","Data":"4e98c0151da94e653d421d37a2d03da09450ddb86ee37d6ed42781ac209d7040"} Jul 15 13:20:29.710573 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:29.710538 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" event={"ID":"eb61a06f-a9d4-4f75-a4c8-64d5b208578f","Type":"ContainerStarted","Data":"425f57d64430ff3fa7527b806be9dfeeaab809130c346551b83a805fcccee360"} Jul 15 13:20:29.728047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:29.728001 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" podStartSLOduration=2.13516761 podStartE2EDuration="2.727985789s" totalImagesPullingTime="592.818179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:20:29.725803196 +0000 UTC m=+844.895749562" watchObservedRunningTime="2026-07-15 13:20:29.727985789 +0000 UTC m=+844.897932195" Jul 15 13:20:30.301248 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:30.301205 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:20:30.816744 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:30.816698 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:20:37.993387 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:37.993348 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:37.993804 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:37.993409 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:38.006024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:38.006000 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:20:38.765384 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:20:38.765356 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5" Jul 15 13:21:19.911154 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:19.911121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:21:20.011556 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:20.011517 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:21:20.114402 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:20.114375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:21:21.150095 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:21.150055 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:21:21.255975 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:21.255943 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:21:25.277604 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:25.277572 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:21:25.279548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:25.279528 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:21:45.388255 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:45.388223 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:21:56.996184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:21:56.996148 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:22:05.086794 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:22:05.086759 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:22:15.987546 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:22:15.987512 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:22:18.408333 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:22:18.408297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:22:27.591293 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:22:27.591255 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:22:33.385637 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:22:33.385586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:22:35.191432 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:22:35.191393 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:22:35.227856 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:22:35.227808 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:22:58.886659 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:22:58.886576 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:23:11.691801 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:11.691766 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:23:16.141350 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:16.141315 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:16.241848 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:16.241816 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:16.346464 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:16.346433 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:22.816227 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:22.816182 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:22.944018 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:22.943992 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:23.096444 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:23.096359 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:23.578421 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:23.578386 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:29.591400 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:29.591358 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:29.663154 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:29.663120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:23:46.485234 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:46.485203 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:23:56.194612 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:23:56.194576 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:24:04.488348 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:04.488312 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:24:04.684019 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:04.683990 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:24:04.798487 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:04.798410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:24:14.792779 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:14.792746 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:24:24.592374 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:24.592341 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:24:33.492380 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:33.492347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:24:43.623852 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:43.623817 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:24:45.381994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:45.381960 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:24:49.334750 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:49.334703 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:24:54.689421 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:24:54.689386 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:25:17.300376 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:25:17.300342 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:25:27.390312 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:25:27.390275 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:25:51.097284 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:25:51.097202 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:26:01.192898 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:26:01.192861 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:26:16.503764 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:26:16.503728 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:26:25.306788 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:26:25.306760 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:26:25.309165 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:26:25.309144 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:26:26.789948 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:26:26.789912 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:26:35.994360 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:26:35.994324 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:26:44.092585 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:26:44.092545 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:26:56.190537 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:26:56.190499 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:27:21.440478 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:21.440444 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn"] Jul 15 13:27:21.441017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:21.440645 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" podUID="30e69da1-fce2-4054-93fe-d0c042e421bc" containerName="manager" containerID="cri-o://574a0c19615836d2243b4f796c84a99fdebd7f3e5b9f447a1a0aac0f06fc83ef" gracePeriod=10 Jul 15 13:27:23.580219 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.580191 2582 generic.go:356] "Generic (PLEG): container finished" podID="30e69da1-fce2-4054-93fe-d0c042e421bc" containerID="574a0c19615836d2243b4f796c84a99fdebd7f3e5b9f447a1a0aac0f06fc83ef" exitCode=0 Jul 15 13:27:23.580554 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.580238 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" event={"ID":"30e69da1-fce2-4054-93fe-d0c042e421bc","Type":"ContainerDied","Data":"574a0c19615836d2243b4f796c84a99fdebd7f3e5b9f447a1a0aac0f06fc83ef"} Jul 15 13:27:23.696336 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.696312 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:27:23.722014 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.721990 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30e69da1-fce2-4054-93fe-d0c042e421bc-extensions-socket-volume\") pod \"30e69da1-fce2-4054-93fe-d0c042e421bc\" (UID: \"30e69da1-fce2-4054-93fe-d0c042e421bc\") " Jul 15 13:27:23.722136 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.722102 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s74cj\" (UniqueName: \"kubernetes.io/projected/30e69da1-fce2-4054-93fe-d0c042e421bc-kube-api-access-s74cj\") pod \"30e69da1-fce2-4054-93fe-d0c042e421bc\" (UID: \"30e69da1-fce2-4054-93fe-d0c042e421bc\") " Jul 15 13:27:23.722354 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.722330 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e69da1-fce2-4054-93fe-d0c042e421bc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "30e69da1-fce2-4054-93fe-d0c042e421bc" (UID: "30e69da1-fce2-4054-93fe-d0c042e421bc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:27:23.724121 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.724093 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e69da1-fce2-4054-93fe-d0c042e421bc-kube-api-access-s74cj" (OuterVolumeSpecName: "kube-api-access-s74cj") pod "30e69da1-fce2-4054-93fe-d0c042e421bc" (UID: "30e69da1-fce2-4054-93fe-d0c042e421bc"). InnerVolumeSpecName "kube-api-access-s74cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:27:23.823376 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.823314 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30e69da1-fce2-4054-93fe-d0c042e421bc-extensions-socket-volume\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:27:23.823376 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:23.823337 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s74cj\" (UniqueName: \"kubernetes.io/projected/30e69da1-fce2-4054-93fe-d0c042e421bc-kube-api-access-s74cj\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:27:24.585589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:24.585561 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" Jul 15 13:27:24.585967 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:24.585555 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn" event={"ID":"30e69da1-fce2-4054-93fe-d0c042e421bc","Type":"ContainerDied","Data":"3bf88af12bb1b67d378ab1b076252fcefef98444f89d7a76eae91679a67ebe6c"} Jul 15 13:27:24.585967 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:24.585691 2582 scope.go:117] "RemoveContainer" containerID="574a0c19615836d2243b4f796c84a99fdebd7f3e5b9f447a1a0aac0f06fc83ef" Jul 15 13:27:24.606085 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:24.606061 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn"] Jul 15 13:27:24.610547 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:24.610527 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sfzhn"] Jul 15 13:27:25.461879 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:27:25.461851 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e69da1-fce2-4054-93fe-d0c042e421bc" path="/var/lib/kubelet/pods/30e69da1-fce2-4054-93fe-d0c042e421bc/volumes" Jul 15 13:28:22.602321 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.602287 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m"] Jul 15 13:28:22.602918 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.602775 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e69da1-fce2-4054-93fe-d0c042e421bc" containerName="manager" Jul 15 13:28:22.602918 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.602794 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e69da1-fce2-4054-93fe-d0c042e421bc" containerName="manager" Jul 15 13:28:22.602918 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.602883 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e69da1-fce2-4054-93fe-d0c042e421bc" containerName="manager" Jul 15 13:28:22.605796 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.605776 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:22.607989 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.607970 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fnrbn\"" Jul 15 13:28:22.617263 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.617242 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m"] Jul 15 13:28:22.766997 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.766970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8db85\" (UniqueName: \"kubernetes.io/projected/3a03ef6b-6857-4629-9bd7-fd0ba3a94931-kube-api-access-8db85\") pod \"kuadrant-operator-controller-manager-86878c769f-87l8m\" (UID: \"3a03ef6b-6857-4629-9bd7-fd0ba3a94931\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:22.767101 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.767072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a03ef6b-6857-4629-9bd7-fd0ba3a94931-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-87l8m\" (UID: \"3a03ef6b-6857-4629-9bd7-fd0ba3a94931\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:22.868285 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.868222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a03ef6b-6857-4629-9bd7-fd0ba3a94931-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-87l8m\" (UID: \"3a03ef6b-6857-4629-9bd7-fd0ba3a94931\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:22.868285 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.868266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8db85\" (UniqueName: \"kubernetes.io/projected/3a03ef6b-6857-4629-9bd7-fd0ba3a94931-kube-api-access-8db85\") pod \"kuadrant-operator-controller-manager-86878c769f-87l8m\" (UID: \"3a03ef6b-6857-4629-9bd7-fd0ba3a94931\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:22.868619 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.868598 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a03ef6b-6857-4629-9bd7-fd0ba3a94931-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-87l8m\" (UID: \"3a03ef6b-6857-4629-9bd7-fd0ba3a94931\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:22.881551 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.881527 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8db85\" (UniqueName: \"kubernetes.io/projected/3a03ef6b-6857-4629-9bd7-fd0ba3a94931-kube-api-access-8db85\") pod \"kuadrant-operator-controller-manager-86878c769f-87l8m\" (UID: \"3a03ef6b-6857-4629-9bd7-fd0ba3a94931\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:22.915362 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:22.915336 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:23.048554 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:23.048525 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m"] Jul 15 13:28:23.052091 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:28:23.052066 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a03ef6b_6857_4629_9bd7_fd0ba3a94931.slice/crio-290b60aeeeba9ac9a289dc5366bfc79e5cfa95d2ed331884e802a29ddb67bf39 WatchSource:0}: Error finding container 290b60aeeeba9ac9a289dc5366bfc79e5cfa95d2ed331884e802a29ddb67bf39: Status 404 returned error can't find the container with id 290b60aeeeba9ac9a289dc5366bfc79e5cfa95d2ed331884e802a29ddb67bf39 Jul 15 13:28:23.837915 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:23.837875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" event={"ID":"3a03ef6b-6857-4629-9bd7-fd0ba3a94931","Type":"ContainerStarted","Data":"930cc44d32577f6f74ccbffeac2d51d83ccae48adf9f6683d97c12b841b34c06"} Jul 15 13:28:23.838400 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:23.837938 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" event={"ID":"3a03ef6b-6857-4629-9bd7-fd0ba3a94931","Type":"ContainerStarted","Data":"290b60aeeeba9ac9a289dc5366bfc79e5cfa95d2ed331884e802a29ddb67bf39"} Jul 15 13:28:23.838400 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:23.837969 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:23.857389 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:23.857339 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" podStartSLOduration=1.8573253090000001 podStartE2EDuration="1.857325309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:28:23.85629021 +0000 UTC m=+1319.026236562" watchObservedRunningTime="2026-07-15 13:28:23.857325309 +0000 UTC m=+1319.027271661" Jul 15 13:28:34.844803 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:34.844770 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-87l8m" Jul 15 13:28:42.590619 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:42.590585 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:28:54.490227 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:28:54.490152 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:29:03.587224 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:29:03.587181 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:29:15.386434 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:29:15.386392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:29:26.681326 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:29:26.681296 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:29:35.691503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:29:35.691468 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:29:45.883136 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:29:45.883105 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:29:55.990415 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:29:55.990371 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:30:07.189030 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:07.189001 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:30:09.324548 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:09.324519 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:30:14.973023 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:14.972985 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:30:15.287911 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:15.287834 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:30:19.763004 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:19.762967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:30:23.886184 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:23.886150 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:30:34.388133 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:34.388097 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:30:50.994579 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:50.994540 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:30:55.563539 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:30:55.563503 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:31:00.294384 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:31:00.294341 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:31:10.989706 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:31:10.989673 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:31:21.500648 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:31:21.500614 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:31:25.333432 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:31:25.333402 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:31:25.337601 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:31:25.337573 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:31:34.578277 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:31:34.578243 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:31:46.890359 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:31:46.890322 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:31:56.392202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:31:56.392165 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:32:06.702354 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:32:06.702319 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:32:15.402094 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:32:15.402055 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:32:24.603374 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:32:24.603338 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:32:38.285590 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:32:38.285548 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:32:44.361405 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:32:44.361373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:32:47.003628 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:32:47.003592 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:32:56.598672 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:32:56.598631 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:33:07.518594 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:33:07.518560 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:33:20.604982 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:33:20.604946 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:33:32.597076 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:33:32.597040 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:33:43.929078 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:33:43.929046 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:33:56.301120 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:33:56.301080 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:34:07.201431 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:07.201396 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:34:12.808531 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:12.808499 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:12.909994 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:12.909966 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:13.233198 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:13.233169 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:19.693885 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.693850 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:19.804473 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.804436 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:19.848474 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.848361 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr"] Jul 15 13:34:19.851693 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.851673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.854417 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.854395 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fb3eae66-openshift-default-dockercfg-27zhc\"" Jul 15 13:34:19.880188 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.880168 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr"] Jul 15 13:34:19.896797 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.896778 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:19.913794 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.913770 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnpxp\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-kube-api-access-gnpxp\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.913879 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.913821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istiod-ca-cert\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.913879 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.913860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-socket\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.913949 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.913928 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-data\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.913987 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.913961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-podinfo\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.914021 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.913989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-envoy\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.914054 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.914035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-credential-socket\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.914087 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.914061 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-certs\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.914121 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.914084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-token\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:19.995674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:19.995645 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:20.015416 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-podinfo\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015520 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-envoy\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015520 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015452 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-credential-socket\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015520 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-certs\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015520 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-token\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015780 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnpxp\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-kube-api-access-gnpxp\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015780 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istiod-ca-cert\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015780 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-socket\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015780 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-data\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.015987 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.015964 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-certs\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.016049 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.016027 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-credential-socket\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.016102 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.016067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-socket\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.016232 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.016218 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-data\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.016519 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.016497 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istiod-ca-cert\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.017782 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.017764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-envoy\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.018106 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.018088 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-podinfo\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.023929 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.023902 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-token\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.024033 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.023961 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnpxp\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-kube-api-access-gnpxp\") pod \"e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.161807 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.161784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:20.298800 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.298776 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr"] Jul 15 13:34:20.301348 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:34:20.301322 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c475814_8ff0_42bf_8ecf_90b39b4eba99.slice/crio-cbca12ed5cd2f9dc67cfe44b7662120421f97b3aca7a868b6d38745b7d2b188b WatchSource:0}: Error finding container cbca12ed5cd2f9dc67cfe44b7662120421f97b3aca7a868b6d38745b7d2b188b: Status 404 returned error can't find the container with id cbca12ed5cd2f9dc67cfe44b7662120421f97b3aca7a868b6d38745b7d2b188b Jul 15 13:34:20.303473 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.303432 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:34:20.303544 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.303525 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:34:20.303584 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.303565 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:34:20.465571 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.465534 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" event={"ID":"2c475814-8ff0-42bf-8ecf-90b39b4eba99","Type":"ContainerStarted","Data":"abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481"} Jul 15 13:34:20.465737 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.465600 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" event={"ID":"2c475814-8ff0-42bf-8ecf-90b39b4eba99","Type":"ContainerStarted","Data":"cbca12ed5cd2f9dc67cfe44b7662120421f97b3aca7a868b6d38745b7d2b188b"} Jul 15 13:34:20.510847 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:20.510793 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" podStartSLOduration=1.5107742320000002 podStartE2EDuration="1.510774232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:34:20.509236737 +0000 UTC m=+1675.679183115" watchObservedRunningTime="2026-07-15 13:34:20.510774232 +0000 UTC m=+1675.680720586" Jul 15 13:34:21.129296 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:21.129265 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:21.162268 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:21.162235 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:21.167068 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:21.167044 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:21.223017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:21.222988 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:21.445024 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:21.444954 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:21.471300 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:21.471259 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:21.472149 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:21.472120 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:21.649337 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:21.649302 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:22.288672 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:22.288635 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:22.399553 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:22.399523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:23.527294 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:23.527248 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:34:28.694935 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:28.694900 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:34:36.558910 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:36.558432 2582 patch_prober.go:28] interesting pod/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": read tcp 10.132.0.2:49514->10.132.0.37:15021: read: connection reset by peer" start-of-body= Jul 15 13:34:36.558910 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:36.558498 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": read tcp 10.132.0.2:49514->10.132.0.37:15021: read: connection reset by peer" Jul 15 13:34:37.550088 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:37.550056 2582 generic.go:356] "Generic (PLEG): container finished" podID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerID="abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481" exitCode=0 Jul 15 13:34:37.550249 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:37.550128 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" event={"ID":"2c475814-8ff0-42bf-8ecf-90b39b4eba99","Type":"ContainerDied","Data":"abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481"} Jul 15 13:34:37.550482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:37.550466 2582 scope.go:117] "RemoveContainer" containerID="abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481" Jul 15 13:34:37.551371 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:37.551348 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:34:37.551444 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:37.551432 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:34:37.551480 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:37.551467 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:34:38.558694 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:38.558659 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" event={"ID":"2c475814-8ff0-42bf-8ecf-90b39b4eba99","Type":"ContainerStarted","Data":"ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972"} Jul 15 13:34:39.162179 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:39.162146 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:39.162382 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:39.162360 2582 patch_prober.go:28] interesting pod/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 15 13:34:39.162448 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:39.162407 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 15 13:34:40.162327 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:40.162291 2582 patch_prober.go:28] interesting pod/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 15 13:34:40.162699 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:40.162349 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 15 13:34:41.162750 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:41.162701 2582 patch_prober.go:28] interesting pod/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 15 13:34:41.163116 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:41.162768 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 15 13:34:42.162517 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:42.162479 2582 patch_prober.go:28] interesting pod/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 15 13:34:42.162690 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:42.162534 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 15 13:34:43.166013 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:43.165983 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:43.166468 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:43.166363 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:34:43.166882 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:34:43.166862 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:35:36.653820 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:36.653787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:39.491228 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:39.491199 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:35:41.200877 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:41.200843 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:47.590953 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:47.590916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:47.714739 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:47.714690 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:51.397387 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:51.397345 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr"] Jul 15 13:35:51.397833 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:51.397531 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" containerID="cri-o://ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972" gracePeriod=30 Jul 15 13:35:51.418140 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:51.418118 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:51.787876 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:51.787841 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:51.886642 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:51.886610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:52.144698 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:52.144669 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:53.314325 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:53.314296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:53.433968 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:53.433933 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:53.730725 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:53.730691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:54.031477 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:54.031400 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:54.276870 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:54.276833 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:35:56.643158 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.643138 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:35:56.785700 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.785666 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-envoy\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.785877 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.785746 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-token\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.785877 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.785777 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-data\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.785877 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.785805 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-certs\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.786083 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786041 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-socket\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.786160 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786083 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:35:56.786160 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786090 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-podinfo\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.786281 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786188 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnpxp\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-kube-api-access-gnpxp\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.786281 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786218 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-credential-socket\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.786281 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786267 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istiod-ca-cert\") pod \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\" (UID: \"2c475814-8ff0-42bf-8ecf-90b39b4eba99\") " Jul 15 13:35:56.786428 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786356 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:35:56.786633 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:35:56.786633 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786565 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-certs\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.786633 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786585 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-workload-socket\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.786633 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786564 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:35:56.786944 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.786920 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:35:56.787945 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.787925 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:35:56.788179 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.788160 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-kube-api-access-gnpxp" (OuterVolumeSpecName: "kube-api-access-gnpxp") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "kube-api-access-gnpxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:35:56.788363 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.788337 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:35:56.788363 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.788333 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c475814-8ff0-42bf-8ecf-90b39b4eba99" (UID: "2c475814-8ff0-42bf-8ecf-90b39b4eba99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:35:56.887963 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.887924 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-envoy\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.887963 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.887959 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-token\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.887963 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.887968 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-data\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.887963 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.887976 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istio-podinfo\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.888215 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.887987 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnpxp\" (UniqueName: \"kubernetes.io/projected/2c475814-8ff0-42bf-8ecf-90b39b4eba99-kube-api-access-gnpxp\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.888215 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.887996 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c475814-8ff0-42bf-8ecf-90b39b4eba99-credential-socket\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.888215 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.888004 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c475814-8ff0-42bf-8ecf-90b39b4eba99-istiod-ca-cert\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:35:56.915487 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.915457 2582 generic.go:356] "Generic (PLEG): container finished" podID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerID="ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972" exitCode=0 Jul 15 13:35:56.915644 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.915520 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" Jul 15 13:35:56.915644 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.915531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" event={"ID":"2c475814-8ff0-42bf-8ecf-90b39b4eba99","Type":"ContainerDied","Data":"ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972"} Jul 15 13:35:56.915644 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.915574 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr" event={"ID":"2c475814-8ff0-42bf-8ecf-90b39b4eba99","Type":"ContainerDied","Data":"cbca12ed5cd2f9dc67cfe44b7662120421f97b3aca7a868b6d38745b7d2b188b"} Jul 15 13:35:56.915644 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.915605 2582 scope.go:117] "RemoveContainer" containerID="ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972" Jul 15 13:35:56.924584 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.924569 2582 scope.go:117] "RemoveContainer" containerID="abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481" Jul 15 13:35:56.932164 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.932142 2582 scope.go:117] "RemoveContainer" containerID="ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972" Jul 15 13:35:56.932471 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:35:56.932448 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972\": container with ID starting with ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972 not found: ID does not exist" containerID="ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972" Jul 15 13:35:56.932562 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.932476 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972"} err="failed to get container status \"ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972\": rpc error: code = NotFound desc = could not find container \"ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972\": container with ID starting with ef2597626b8f548951ff58fc7da3edee1857a12bd57497bd460c58f026cab972 not found: ID does not exist" Jul 15 13:35:56.932562 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.932493 2582 scope.go:117] "RemoveContainer" containerID="abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481" Jul 15 13:35:56.932772 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:35:56.932755 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481\": container with ID starting with abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481 not found: ID does not exist" containerID="abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481" Jul 15 13:35:56.932826 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.932777 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481"} err="failed to get container status \"abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481\": rpc error: code = NotFound desc = could not find container \"abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481\": container with ID starting with abbfcc9b9591d6d960b0bc765cd295d85e1bba37dc86e954a3829a5ddefc6481 not found: ID does not exist" Jul 15 13:35:56.935759 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.935735 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr"] Jul 15 13:35:56.939144 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:56.939123 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fb3eae66-openshift-default-595fb5d567-r9dpr"] Jul 15 13:35:57.461612 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:35:57.461578 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" path="/var/lib/kubelet/pods/2c475814-8ff0-42bf-8ecf-90b39b4eba99/volumes" Jul 15 13:36:00.496503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:00.496472 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:36:03.134349 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:03.134322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:36:03.194560 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:03.194527 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:36:06.655199 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:06.655166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:36:10.189311 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:10.189226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:36:10.290073 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:10.290042 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:36:25.360157 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:25.360124 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:36:25.364659 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:25.364634 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:36:57.954959 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:57.954924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:36:58.054515 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:58.054485 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:36:58.483263 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:36:58.483221 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:00.589863 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:00.589823 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:37:04.176218 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:04.176189 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:04.242915 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:04.242882 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:04.386114 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:04.386084 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:04.609697 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:04.609670 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:04.676536 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:04.676490 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:05.522570 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:05.522537 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:05.695735 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:05.695690 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:05.811604 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:05.811527 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:06.115368 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:06.115293 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:06.359327 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:06.359288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:06.933155 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:06.933119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:08.074006 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:08.073979 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:09.490288 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:09.490254 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:37:57.680697 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:57.680609 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:37:57.772398 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:37:57.772363 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:01.260911 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:01.260876 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:01.392064 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:01.392036 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:11.124170 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:11.124122 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:13.585238 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:13.585200 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:38:15.721588 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:15.721557 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:20.102264 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:20.102225 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:20.203522 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:20.203485 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:22.462791 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:22.462755 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:22.563948 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:22.563921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:26.740902 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:26.740867 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:27.777179 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:27.777150 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:28.123025 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:28.122938 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:28.214296 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:28.214263 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:28.371222 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:28.371187 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:28.475946 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:28.475917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:28.996954 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:28.996913 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:29.094755 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:29.094702 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:30.788060 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:30.788028 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:38:33.146165 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:33.146126 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:34.036335 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:34.036296 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:38:36.425699 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:38:36.425670 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:39:43.046827 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:43.046794 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:39:45.986880 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:45.986846 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:39:47.707215 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:47.707182 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:39:56.534428 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:56.534393 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:39:56.604267 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:56.604234 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:39:57.885174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:57.885137 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:39:58.502403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:58.502366 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:39:58.601414 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:58.601385 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:39:58.701792 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:39:58.701752 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:00.108261 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:00.108230 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:04.412067 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:04.412031 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:06.115464 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:06.115434 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:06.216410 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:06.216380 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:06.315310 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:06.315277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:06.416693 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:06.416612 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:06.808647 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:06.808617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:32.771340 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:32.771307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:38.917556 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:38.917523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:47.573413 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:47.573332 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:40:55.026860 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:40:55.026814 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:01.377706 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:01.377670 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:01.485142 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:01.485114 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:01.578658 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:01.578632 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:12.115506 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:12.115473 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:12.902161 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:12.902130 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:18.925627 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:18.925597 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:19.004501 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:19.004467 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:19.082219 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:19.082191 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:19.184489 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:19.184429 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:20.015883 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:20.015845 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:20.116449 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:20.116419 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:20.216619 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:20.216592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:21.301487 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:21.301451 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:41:21.832811 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:21.832772 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:25.392397 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:25.392368 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:41:25.398574 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:25.398556 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:41:28.007685 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:28.007654 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:28.077186 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:28.077152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:28.717180 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:28.717150 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:28.798296 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:28.798263 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:41:35.127167 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:35.127136 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:35.197422 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:35.197393 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:35.335925 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:35.335896 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:35.659158 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:35.659123 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:35.758430 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:35.758377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:37.685428 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:37.685395 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:41:38.673517 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:38.673483 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:38.762604 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:38.762576 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:38.942094 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:38.941998 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:39.136616 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:39.136586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:39.432490 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:39.432457 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:39.533027 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:39.532997 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:43.187466 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:43.187431 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:43.814887 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:43.814850 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:43.907177 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:43.907141 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:44.260162 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:44.260128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:47.985918 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:47.985883 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:41:50.433731 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:50.433686 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:41:50.534611 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:41:50.534583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:01.260154 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:01.260125 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:03.990591 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:03.990558 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:42:05.876325 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:05.876290 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:11.828036 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:11.828004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:11.928009 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:11.927980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:16.314651 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:16.314618 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:16.575033 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:16.574952 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:16.824903 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:16.824869 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:16.950900 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:16.950813 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:17.050480 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:17.050452 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:17.184648 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:17.184580 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:25.748320 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:25.748286 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:25.849508 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:25.849474 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:29.373975 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:29.373942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:30.098666 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:30.098633 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:36.227560 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:36.227522 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:36.524025 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:36.523938 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:36.724849 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:36.724811 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:38.997443 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:38.997414 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:39.089685 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:39.089655 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:39.157030 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:39.157001 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:39.588755 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:39.588706 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:39.780874 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:39.780847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:39.889037 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:39.888966 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:43.520116 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:43.520084 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:44.661641 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:44.661607 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:45.989456 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:45.989422 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:42:51.110692 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:51.110665 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:55.393845 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:55.393808 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:42:58.475959 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:58.475925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:42:58.837275 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:42:58.837191 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:43:00.636005 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:43:00.635974 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:43:00.734192 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:43:00.734152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:43:18.285604 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:43:18.285574 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:43:32.296199 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:43:32.296167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:43:36.391321 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:43:36.391278 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:43:36.914671 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:43:36.914636 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:43:41.947062 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:43:41.947021 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:18.699759 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:18.699647 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:18.883388 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:18.883361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:19.420160 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:19.420126 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:25.188553 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:25.188520 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:25.245950 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:25.245924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:29.194990 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:29.194958 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:29.410780 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:29.410746 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:30.458732 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:30.458690 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:30.557532 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:30.557505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:30.657853 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:30.657819 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:31.427206 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:31.427176 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:31.510647 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:31.510600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:31.610205 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:31.610175 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:33.054318 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:33.054286 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:33.091806 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:33.091780 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:44:39.269390 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:39.269355 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:39.370164 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:39.370130 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:39.972632 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:39.972601 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:44.491551 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:44.491516 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:44:45.965874 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:45.965846 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:46.027352 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:46.027326 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:46.089431 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:46.089408 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:46.192403 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:46.192377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:56.694970 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:56.694931 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:44:57.063738 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:44:57.063686 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:18.001202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:18.001120 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:24.000260 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:24.000226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:24.068013 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:24.067986 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:24.415227 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:24.415145 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:24.717560 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:24.717507 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:25.285884 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:25.285853 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:27.952522 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:27.952482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:28.030109 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:28.030062 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:28.092933 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:28.092882 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:28.473981 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:28.473948 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:28.615566 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:28.615531 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:28.716326 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:28.716294 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:32.468809 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:32.468777 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:32.791538 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:32.791442 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:33.101317 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:33.101206 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:33.252056 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:33.252016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:35.288425 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:35.288383 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:45:36.013133 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:36.013098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:36.113997 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:36.113966 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:36.211739 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:36.211695 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:36.335243 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:36.335161 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:36.686422 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:36.686295 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:36.788705 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:36.788666 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:40.550981 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:40.550941 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:46.189446 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:46.189416 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:45:49.390856 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:49.390821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:50.789240 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.789208 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97"] Jul 15 13:45:50.789598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.789515 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" Jul 15 13:45:50.789598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.789526 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" Jul 15 13:45:50.789598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.789538 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" Jul 15 13:45:50.789598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.789544 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" Jul 15 13:45:50.789762 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.789616 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" Jul 15 13:45:50.792227 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.792207 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:50.794391 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.794370 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-486f14\"/\"kube-root-ca.crt\"" Jul 15 13:45:50.794948 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.794926 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-486f14\"/\"default-dockercfg-jzclp\"" Jul 15 13:45:50.795046 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.794926 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-486f14\"/\"test-model-486f14-kserve-self-signed-certs\"" Jul 15 13:45:50.795046 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.794932 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-486f14\"/\"openshift-service-ca.crt\"" Jul 15 13:45:50.804803 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.804784 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97"] Jul 15 13:45:50.826101 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.826080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:50.901310 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.901281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-home\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:50.901443 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.901320 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw7hx\" (UniqueName: \"kubernetes.io/projected/3d2d3456-789a-400d-95d7-304bed4e87b3-kube-api-access-tw7hx\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:50.901443 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.901337 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d3456-789a-400d-95d7-304bed4e87b3-tls-certs\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:50.901443 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.901394 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-dshm\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:50.901443 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.901424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-tmp-dir\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:50.901598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.901485 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-model-cache\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:50.925479 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:50.925454 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:51.002487 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.002461 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw7hx\" (UniqueName: \"kubernetes.io/projected/3d2d3456-789a-400d-95d7-304bed4e87b3-kube-api-access-tw7hx\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.002632 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.002491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d3456-789a-400d-95d7-304bed4e87b3-tls-certs\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.002632 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.002529 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-dshm\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.002632 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.002571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-tmp-dir\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.002788 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.002632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-model-cache\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.002788 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.002703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-home\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.003037 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.003017 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-tmp-dir\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.003128 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.003067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-home\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.003196 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.003130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-model-cache\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.005187 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.005160 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-dshm\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.005289 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.005275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d3456-789a-400d-95d7-304bed4e87b3-tls-certs\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.009671 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.009651 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw7hx\" (UniqueName: \"kubernetes.io/projected/3d2d3456-789a-400d-95d7-304bed4e87b3-kube-api-access-tw7hx\") pod \"test-model-486f14-kserve-6c8bf8dd8c-rgk97\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.101914 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.101856 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:45:51.235905 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.235879 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97"] Jul 15 13:45:51.238097 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:45:51.238064 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d2d3456_789a_400d_95d7_304bed4e87b3.slice/crio-4fbe1226a2cd5bce01333bda1ba121b865d2ab6fcac495a4dd760286b608f02c WatchSource:0}: Error finding container 4fbe1226a2cd5bce01333bda1ba121b865d2ab6fcac495a4dd760286b608f02c: Status 404 returned error can't find the container with id 4fbe1226a2cd5bce01333bda1ba121b865d2ab6fcac495a4dd760286b608f02c Jul 15 13:45:51.573741 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.573682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" event={"ID":"3d2d3456-789a-400d-95d7-304bed4e87b3","Type":"ContainerStarted","Data":"29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d"} Jul 15 13:45:51.573942 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.573756 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" event={"ID":"3d2d3456-789a-400d-95d7-304bed4e87b3","Type":"ContainerStarted","Data":"4fbe1226a2cd5bce01333bda1ba121b865d2ab6fcac495a4dd760286b608f02c"} Jul 15 13:45:51.591323 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:51.591274 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" podStartSLOduration=1.5912589339999998 podStartE2EDuration="1.591258934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:45:51.58963458 +0000 UTC m=+2366.759580961" watchObservedRunningTime="2026-07-15 13:45:51.591258934 +0000 UTC m=+2366.761205287" Jul 15 13:45:53.044401 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:53.044366 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:53.144825 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:53.144793 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:53.373215 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:53.373135 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:54.394016 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:54.393981 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:45:57.573000 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:57.572961 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:45:57.675852 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:45:57.675822 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:01.102490 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:01.102459 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:46:01.102490 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:01.102499 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:46:01.115150 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:01.115125 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:46:01.628057 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:01.628025 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:46:19.594967 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.594932 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn"] Jul 15 13:46:19.595357 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.595292 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c475814-8ff0-42bf-8ecf-90b39b4eba99" containerName="istio-proxy" Jul 15 13:46:19.598271 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.598251 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.601097 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.601074 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a5d1bc\"/\"kube-root-ca.crt\"" Jul 15 13:46:19.601214 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.601109 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a5d1bc\"/\"openshift-service-ca.crt\"" Jul 15 13:46:19.601214 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.601121 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a5d1bc\"/\"default-dockercfg-jp7l5\"" Jul 15 13:46:19.601214 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.601147 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a5d1bc\"/\"test-model-a5d1bc-kserve-self-signed-certs\"" Jul 15 13:46:19.608313 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.608292 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn"] Jul 15 13:46:19.692874 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.692836 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:19.716886 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.716859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0840f0f0-7822-4073-9746-cb2d40b0a571-tls-certs\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.717003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.716913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-tmp-dir\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.717003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.716932 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxg5\" (UniqueName: \"kubernetes.io/projected/0840f0f0-7822-4073-9746-cb2d40b0a571-kube-api-access-dxxg5\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.717003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.716953 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-model-cache\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.717003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.716975 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-home\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.717003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.716999 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-dshm\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818133 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-tmp-dir\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818265 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxg5\" (UniqueName: \"kubernetes.io/projected/0840f0f0-7822-4073-9746-cb2d40b0a571-kube-api-access-dxxg5\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818265 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-model-cache\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818265 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818209 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-home\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818265 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-dshm\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818499 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818287 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0840f0f0-7822-4073-9746-cb2d40b0a571-tls-certs\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818499 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-tmp-dir\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818611 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-model-cache\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.818668 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.818609 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-home\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.820564 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.820541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-dshm\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.820677 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.820657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0840f0f0-7822-4073-9746-cb2d40b0a571-tls-certs\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.825488 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.825470 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxg5\" (UniqueName: \"kubernetes.io/projected/0840f0f0-7822-4073-9746-cb2d40b0a571-kube-api-access-dxxg5\") pod \"test-model-a5d1bc-kserve-74474cb785-jq7nn\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:19.908435 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:19.908362 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:20.037996 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:20.037970 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn"] Jul 15 13:46:20.040181 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:46:20.040146 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0840f0f0_7822_4073_9746_cb2d40b0a571.slice/crio-528aa09eb12a66c31e5d06679a511eae501692434660934b5a178bc185fc69ca WatchSource:0}: Error finding container 528aa09eb12a66c31e5d06679a511eae501692434660934b5a178bc185fc69ca: Status 404 returned error can't find the container with id 528aa09eb12a66c31e5d06679a511eae501692434660934b5a178bc185fc69ca Jul 15 13:46:20.701805 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:20.701770 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" event={"ID":"0840f0f0-7822-4073-9746-cb2d40b0a571","Type":"ContainerStarted","Data":"0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9"} Jul 15 13:46:20.702174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:20.701812 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" event={"ID":"0840f0f0-7822-4073-9746-cb2d40b0a571","Type":"ContainerStarted","Data":"528aa09eb12a66c31e5d06679a511eae501692434660934b5a178bc185fc69ca"} Jul 15 13:46:20.720621 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:20.720575 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" podStartSLOduration=1.720558782 podStartE2EDuration="1.720558782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:46:20.71878748 +0000 UTC m=+2395.888733845" watchObservedRunningTime="2026-07-15 13:46:20.720558782 +0000 UTC m=+2395.890505134" Jul 15 13:46:23.992744 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:23.992296 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:46:25.418598 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:25.418565 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:46:25.424781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:25.424757 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:46:29.908843 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:29.908810 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:29.908843 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:29.908848 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:29.921121 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:29.921099 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:30.763518 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:30.763491 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:46:35.040130 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:35.040098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:35.140392 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:35.140361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:37.989021 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:37.988985 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:46:44.190738 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:44.190679 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97"] Jul 15 13:46:44.210320 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:44.210276 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:45.142732 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:45.142686 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn"] Jul 15 13:46:45.156990 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:45.156958 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:45.778677 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:45.778642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:45.879698 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:45.879611 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:50.290517 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:50.290437 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:46:55.852099 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:55.852053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:46:59.210946 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.210888 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" podUID="3d2d3456-789a-400d-95d7-304bed4e87b3" containerName="main" containerID="cri-o://29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d" gracePeriod=45 Jul 15 13:46:59.471778 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.471705 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:46:59.545674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.545646 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-tmp-dir\") pod \"3d2d3456-789a-400d-95d7-304bed4e87b3\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " Jul 15 13:46:59.545859 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.545689 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-model-cache\") pod \"3d2d3456-789a-400d-95d7-304bed4e87b3\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " Jul 15 13:46:59.545859 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.545750 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-dshm\") pod \"3d2d3456-789a-400d-95d7-304bed4e87b3\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " Jul 15 13:46:59.545859 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.545789 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-home\") pod \"3d2d3456-789a-400d-95d7-304bed4e87b3\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " Jul 15 13:46:59.545859 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.545819 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d3456-789a-400d-95d7-304bed4e87b3-tls-certs\") pod \"3d2d3456-789a-400d-95d7-304bed4e87b3\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " Jul 15 13:46:59.545859 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.545846 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw7hx\" (UniqueName: \"kubernetes.io/projected/3d2d3456-789a-400d-95d7-304bed4e87b3-kube-api-access-tw7hx\") pod \"3d2d3456-789a-400d-95d7-304bed4e87b3\" (UID: \"3d2d3456-789a-400d-95d7-304bed4e87b3\") " Jul 15 13:46:59.546149 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.545926 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d2d3456-789a-400d-95d7-304bed4e87b3" (UID: "3d2d3456-789a-400d-95d7-304bed4e87b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:46:59.546149 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.546022 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d2d3456-789a-400d-95d7-304bed4e87b3" (UID: "3d2d3456-789a-400d-95d7-304bed4e87b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:46:59.546149 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.546070 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-home" (OuterVolumeSpecName: "home") pod "3d2d3456-789a-400d-95d7-304bed4e87b3" (UID: "3d2d3456-789a-400d-95d7-304bed4e87b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:46:59.546255 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.546202 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-tmp-dir\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:46:59.546255 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.546220 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-model-cache\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:46:59.546255 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.546228 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-home\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:46:59.548057 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.548032 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2d3456-789a-400d-95d7-304bed4e87b3-kube-api-access-tw7hx" (OuterVolumeSpecName: "kube-api-access-tw7hx") pod "3d2d3456-789a-400d-95d7-304bed4e87b3" (UID: "3d2d3456-789a-400d-95d7-304bed4e87b3"). InnerVolumeSpecName "kube-api-access-tw7hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:46:59.548168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.548071 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-dshm" (OuterVolumeSpecName: "dshm") pod "3d2d3456-789a-400d-95d7-304bed4e87b3" (UID: "3d2d3456-789a-400d-95d7-304bed4e87b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:46:59.548168 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.548075 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2d3456-789a-400d-95d7-304bed4e87b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d2d3456-789a-400d-95d7-304bed4e87b3" (UID: "3d2d3456-789a-400d-95d7-304bed4e87b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:46:59.647246 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.647217 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d2d3456-789a-400d-95d7-304bed4e87b3-dshm\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:46:59.647246 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.647240 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d2d3456-789a-400d-95d7-304bed4e87b3-tls-certs\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:46:59.647394 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.647251 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw7hx\" (UniqueName: \"kubernetes.io/projected/3d2d3456-789a-400d-95d7-304bed4e87b3-kube-api-access-tw7hx\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:46:59.887750 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.887662 2582 generic.go:356] "Generic (PLEG): container finished" podID="3d2d3456-789a-400d-95d7-304bed4e87b3" containerID="29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d" exitCode=0 Jul 15 13:46:59.887750 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.887704 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" Jul 15 13:46:59.887900 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.887743 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" event={"ID":"3d2d3456-789a-400d-95d7-304bed4e87b3","Type":"ContainerDied","Data":"29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d"} Jul 15 13:46:59.887900 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.887784 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97" event={"ID":"3d2d3456-789a-400d-95d7-304bed4e87b3","Type":"ContainerDied","Data":"4fbe1226a2cd5bce01333bda1ba121b865d2ab6fcac495a4dd760286b608f02c"} Jul 15 13:46:59.887900 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.887802 2582 scope.go:117] "RemoveContainer" containerID="29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d" Jul 15 13:46:59.897064 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.897049 2582 scope.go:117] "RemoveContainer" containerID="29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d" Jul 15 13:46:59.897304 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:46:59.897287 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d\": container with ID starting with 29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d not found: ID does not exist" containerID="29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d" Jul 15 13:46:59.897349 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.897312 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d"} err="failed to get container status \"29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d\": rpc error: code = NotFound desc = could not find container \"29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d\": container with ID starting with 29c7fe67a0da85d339ed6250ce1b3744a778f3c6e37188010bf1b3c1a9b25a5d not found: ID does not exist" Jul 15 13:46:59.908165 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.908144 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97"] Jul 15 13:46:59.911938 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:46:59.911918 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-486f14/test-model-486f14-kserve-6c8bf8dd8c-rgk97"] Jul 15 13:47:00.163041 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.162959 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" podUID="0840f0f0-7822-4073-9746-cb2d40b0a571" containerName="main" containerID="cri-o://0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9" gracePeriod=45 Jul 15 13:47:00.361691 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.361658 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:00.409798 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.409775 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:47:00.452880 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.452857 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-home\") pod \"0840f0f0-7822-4073-9746-cb2d40b0a571\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " Jul 15 13:47:00.453005 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.452906 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxxg5\" (UniqueName: \"kubernetes.io/projected/0840f0f0-7822-4073-9746-cb2d40b0a571-kube-api-access-dxxg5\") pod \"0840f0f0-7822-4073-9746-cb2d40b0a571\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " Jul 15 13:47:00.453005 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.452924 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-model-cache\") pod \"0840f0f0-7822-4073-9746-cb2d40b0a571\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " Jul 15 13:47:00.453005 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.452946 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-dshm\") pod \"0840f0f0-7822-4073-9746-cb2d40b0a571\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " Jul 15 13:47:00.453174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.453037 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-tmp-dir\") pod \"0840f0f0-7822-4073-9746-cb2d40b0a571\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " Jul 15 13:47:00.453174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.453077 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0840f0f0-7822-4073-9746-cb2d40b0a571-tls-certs\") pod \"0840f0f0-7822-4073-9746-cb2d40b0a571\" (UID: \"0840f0f0-7822-4073-9746-cb2d40b0a571\") " Jul 15 13:47:00.453174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.453126 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-home" (OuterVolumeSpecName: "home") pod "0840f0f0-7822-4073-9746-cb2d40b0a571" (UID: "0840f0f0-7822-4073-9746-cb2d40b0a571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:00.453334 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.453281 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-model-cache" (OuterVolumeSpecName: "model-cache") pod "0840f0f0-7822-4073-9746-cb2d40b0a571" (UID: "0840f0f0-7822-4073-9746-cb2d40b0a571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:00.453334 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.453323 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0840f0f0-7822-4073-9746-cb2d40b0a571" (UID: "0840f0f0-7822-4073-9746-cb2d40b0a571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:00.453438 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.453408 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-tmp-dir\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:47:00.453438 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.453432 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-home\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:47:00.453543 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.453446 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-model-cache\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:47:00.455115 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.455094 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-dshm" (OuterVolumeSpecName: "dshm") pod "0840f0f0-7822-4073-9746-cb2d40b0a571" (UID: "0840f0f0-7822-4073-9746-cb2d40b0a571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:00.455264 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.455246 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0840f0f0-7822-4073-9746-cb2d40b0a571-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0840f0f0-7822-4073-9746-cb2d40b0a571" (UID: "0840f0f0-7822-4073-9746-cb2d40b0a571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:47:00.455324 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.455212 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0840f0f0-7822-4073-9746-cb2d40b0a571-kube-api-access-dxxg5" (OuterVolumeSpecName: "kube-api-access-dxxg5") pod "0840f0f0-7822-4073-9746-cb2d40b0a571" (UID: "0840f0f0-7822-4073-9746-cb2d40b0a571"). InnerVolumeSpecName "kube-api-access-dxxg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:47:00.492988 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.492962 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:47:00.553836 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.553812 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxxg5\" (UniqueName: \"kubernetes.io/projected/0840f0f0-7822-4073-9746-cb2d40b0a571-kube-api-access-dxxg5\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:47:00.553836 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.553832 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0840f0f0-7822-4073-9746-cb2d40b0a571-dshm\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:47:00.553965 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.553841 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0840f0f0-7822-4073-9746-cb2d40b0a571-tls-certs\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:47:00.894020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.893938 2582 generic.go:356] "Generic (PLEG): container finished" podID="0840f0f0-7822-4073-9746-cb2d40b0a571" containerID="0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9" exitCode=0 Jul 15 13:47:00.894020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.893978 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" event={"ID":"0840f0f0-7822-4073-9746-cb2d40b0a571","Type":"ContainerDied","Data":"0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9"} Jul 15 13:47:00.894020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.893982 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" Jul 15 13:47:00.894020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.894007 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn" event={"ID":"0840f0f0-7822-4073-9746-cb2d40b0a571","Type":"ContainerDied","Data":"528aa09eb12a66c31e5d06679a511eae501692434660934b5a178bc185fc69ca"} Jul 15 13:47:00.894306 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.894025 2582 scope.go:117] "RemoveContainer" containerID="0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9" Jul 15 13:47:00.903413 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.903396 2582 scope.go:117] "RemoveContainer" containerID="0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9" Jul 15 13:47:00.903628 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:47:00.903613 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9\": container with ID starting with 0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9 not found: ID does not exist" containerID="0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9" Jul 15 13:47:00.903676 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.903637 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9"} err="failed to get container status \"0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9\": rpc error: code = NotFound desc = could not find container \"0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9\": container with ID starting with 0f8205c81b64f712953e137a6f1ab4ff7240e4e72f63cd3c912e6da66fe123e9 not found: ID does not exist" Jul 15 13:47:00.913857 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.913836 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn"] Jul 15 13:47:00.919595 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:00.919572 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a5d1bc/test-model-a5d1bc-kserve-74474cb785-jq7nn"] Jul 15 13:47:01.460782 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:01.460749 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0840f0f0-7822-4073-9746-cb2d40b0a571" path="/var/lib/kubelet/pods/0840f0f0-7822-4073-9746-cb2d40b0a571/volumes" Jul 15 13:47:01.461138 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:01.461120 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2d3456-789a-400d-95d7-304bed4e87b3" path="/var/lib/kubelet/pods/3d2d3456-789a-400d-95d7-304bed4e87b3/volumes" Jul 15 13:47:06.678582 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:06.678550 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:06.779682 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:06.779645 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:11.040865 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:11.040806 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:11.269208 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:11.269178 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:12.013464 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:12.013418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:12.114980 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:12.114949 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:21.958618 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:21.958581 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:25.086845 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:25.086809 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:47:26.476618 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:26.476583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:30.402196 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:30.402158 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:32.616804 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:32.616770 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:32.682905 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:32.682855 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:37.150661 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:37.150627 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:37.375176 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:37.375135 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:37.696043 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:37.696008 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:38.019062 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:38.019029 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:38.358110 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:38.358027 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:40.623588 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:40.623549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:40.722513 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:40.722472 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:40.816210 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:40.816173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:40.908463 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:40.908356 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:41.656871 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:41.656836 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:41.755676 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:41.755645 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:45.164815 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:45.164785 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:45.588257 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:45.588224 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:47:45.596116 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:45.596088 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:45.893068 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:45.892951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:45.988550 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:45.988520 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:48.700186 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:48.700107 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:48.799147 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:48.799095 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:48.894265 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:48.894233 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:49.049707 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:49.049670 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:49.520047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:49.520013 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:49.620542 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:49.620479 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:53.209209 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:53.209170 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:55.991014 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:55.990981 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:47:59.275442 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:59.275407 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:47:59.351969 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:47:59.351937 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:01.858674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:01.858641 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:01.957464 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:01.957432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:10.315428 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:10.315392 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:10.423104 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:10.423077 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:11.860248 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:11.860215 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:11.961244 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:11.961213 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:12.125551 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:12.125471 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:15.192792 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:15.192760 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:48:16.297590 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:16.297550 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:22.148515 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:22.148482 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:22.214897 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:22.214869 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:26.984038 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:26.984001 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:27.203197 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:27.203166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:27.843262 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:27.843222 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:27.943611 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:27.943576 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:37.829946 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:37.829910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:40.984861 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:40.984827 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:48:42.410554 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:42.410520 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:48.395289 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:48.395256 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:48.495935 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:48.495902 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:53.120915 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:53.120876 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:53.351578 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:53.351546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:53.679739 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:53.679684 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:54.002469 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:54.002433 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:54.962907 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:54.962875 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:57.088440 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:57.088406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:57.196743 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:57.196688 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:58.312385 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:58.312350 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:48:58.370617 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:48:58.370586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:01.489437 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:01.489401 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:49:01.580801 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:01.580767 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:01.956519 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:01.956491 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:02.236632 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:02.236546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:02.535896 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:02.535821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:05.191172 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:05.191129 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:05.282231 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:05.282198 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:05.451702 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:05.451678 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:05.859537 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:05.859461 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:06.017936 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:06.017908 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:06.115937 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:06.115861 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:09.641385 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:09.641356 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:12.281814 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:12.281783 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:49:18.625091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:18.625059 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:18.725093 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:18.725063 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:20.290893 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.290861 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c"] Jul 15 13:49:20.291245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.291168 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0840f0f0-7822-4073-9746-cb2d40b0a571" containerName="main" Jul 15 13:49:20.291245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.291180 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0840f0f0-7822-4073-9746-cb2d40b0a571" containerName="main" Jul 15 13:49:20.291245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.291201 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2d3456-789a-400d-95d7-304bed4e87b3" containerName="main" Jul 15 13:49:20.291245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.291206 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2d3456-789a-400d-95d7-304bed4e87b3" containerName="main" Jul 15 13:49:20.291387 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.291279 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2d3456-789a-400d-95d7-304bed4e87b3" containerName="main" Jul 15 13:49:20.291387 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.291289 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0840f0f0-7822-4073-9746-cb2d40b0a571" containerName="main" Jul 15 13:49:20.294262 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.294247 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.297010 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.296964 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7abf62\"/\"default-dockercfg-q5dv2\"" Jul 15 13:49:20.297010 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.296984 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7abf62\"/\"kube-root-ca.crt\"" Jul 15 13:49:20.297010 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.296971 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7abf62\"/\"openshift-service-ca.crt\"" Jul 15 13:49:20.297238 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.297025 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7abf62\"/\"auth-test-model-7abf62-kserve-self-signed-certs\"" Jul 15 13:49:20.303233 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.303209 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c"] Jul 15 13:49:20.401597 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.401569 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:20.432457 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.432426 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/60db327b-e4f3-46cc-a009-08ea6e2e1046-kube-api-access-6rqjx\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.432621 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.432478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-tmp-dir\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.432621 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.432523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60db327b-e4f3-46cc-a009-08ea6e2e1046-tls-certs\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.432772 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.432629 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-dshm\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.432772 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.432700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-home\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.432889 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.432780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-model-cache\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.533837 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.533810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-dshm\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.533982 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.533856 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-home\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.533982 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.533898 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-model-cache\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.533982 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.533922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/60db327b-e4f3-46cc-a009-08ea6e2e1046-kube-api-access-6rqjx\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.534147 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.534058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-tmp-dir\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.534147 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.534133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60db327b-e4f3-46cc-a009-08ea6e2e1046-tls-certs\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.534368 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.534329 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-home\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.534472 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.534374 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-model-cache\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.534472 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.534420 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-tmp-dir\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.536185 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.536162 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-dshm\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.536581 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.536562 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60db327b-e4f3-46cc-a009-08ea6e2e1046-tls-certs\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.541412 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.541360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/60db327b-e4f3-46cc-a009-08ea6e2e1046-kube-api-access-6rqjx\") pod \"auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.604047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.604013 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:20.733982 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:20.733920 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c"] Jul 15 13:49:20.736778 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:49:20.736747 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60db327b_e4f3_46cc_a009_08ea6e2e1046.slice/crio-06cd68036543d2da10ff76f95e7cd10a56a7a9e8e24b18f13629dd25e879f343 WatchSource:0}: Error finding container 06cd68036543d2da10ff76f95e7cd10a56a7a9e8e24b18f13629dd25e879f343: Status 404 returned error can't find the container with id 06cd68036543d2da10ff76f95e7cd10a56a7a9e8e24b18f13629dd25e879f343 Jul 15 13:49:21.533243 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:21.533202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" event={"ID":"60db327b-e4f3-46cc-a009-08ea6e2e1046","Type":"ContainerStarted","Data":"0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814"} Jul 15 13:49:21.533243 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:21.533247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" event={"ID":"60db327b-e4f3-46cc-a009-08ea6e2e1046","Type":"ContainerStarted","Data":"06cd68036543d2da10ff76f95e7cd10a56a7a9e8e24b18f13629dd25e879f343"} Jul 15 13:49:21.556250 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:21.556187 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" podStartSLOduration=1.556164838 podStartE2EDuration="1.556164838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:49:21.554188294 +0000 UTC m=+2576.724134659" watchObservedRunningTime="2026-07-15 13:49:21.556164838 +0000 UTC m=+2576.726111195" Jul 15 13:49:23.992283 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:23.992250 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:49:27.760214 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:27.760180 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:27.857914 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:27.857883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:30.604135 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:30.604109 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:30.604511 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:30.604365 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:30.616753 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:30.616709 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:31.588473 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:31.588448 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:49:49.027302 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:49.027223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:49:52.289147 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:49:52.289116 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:50:16.395935 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:50:16.395896 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:50:30.281870 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:50:30.281834 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:50:44.792318 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:50:44.792285 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:51:02.183754 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:51:02.183705 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:51:16.894357 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:51:16.894317 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:51:25.445013 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:51:25.444986 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:51:25.451088 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:51:25.451066 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:51:32.896743 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:51:32.896695 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:51:48.592128 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:51:48.592095 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:52:01.568956 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:01.568919 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c"] Jul 15 13:52:01.615562 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:01.615532 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:52:03.613041 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:03.613012 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:52:05.379222 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:05.379194 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:52:06.788029 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.787997 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p"] Jul 15 13:52:06.791201 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.791182 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.793509 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.793490 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7abf62\"/\"sub-test-model-7abf62-kserve-self-signed-certs\"" Jul 15 13:52:06.801447 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.801428 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p"] Jul 15 13:52:06.810289 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.810266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:52:06.881545 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.881516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-tmp-dir\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.881665 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.881548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-dshm\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.881665 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.881627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f14cb6b-081e-4412-8b84-196fd0c45561-tls-certs\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.881665 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.881651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-model-cache\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.881807 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.881708 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-home\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.881807 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.881766 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgj4\" (UniqueName: \"kubernetes.io/projected/8f14cb6b-081e-4412-8b84-196fd0c45561-kube-api-access-fvgj4\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.982994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-tmp-dir\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983165 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.983022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-dshm\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983165 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.983056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f14cb6b-081e-4412-8b84-196fd0c45561-tls-certs\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983165 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.983085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-model-cache\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983333 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.983295 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-home\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983407 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.983384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgj4\" (UniqueName: \"kubernetes.io/projected/8f14cb6b-081e-4412-8b84-196fd0c45561-kube-api-access-fvgj4\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983468 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.983452 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-tmp-dir\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983619 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.983596 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-model-cache\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.983696 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.983639 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-home\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.985585 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.985565 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f14cb6b-081e-4412-8b84-196fd0c45561-tls-certs\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.985781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.985763 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-dshm\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:06.992063 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:06.992034 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgj4\" (UniqueName: \"kubernetes.io/projected/8f14cb6b-081e-4412-8b84-196fd0c45561-kube-api-access-fvgj4\") pod \"sub-test-model-7abf62-kserve-5bb89998f9-4zw8p\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:07.100554 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:07.100500 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:07.236398 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:07.236372 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p"] Jul 15 13:52:07.238466 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:52:07.238435 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f14cb6b_081e_4412_8b84_196fd0c45561.slice/crio-a4e557b00b24557e3a5af866ac437338816202f06f3d2832eff4b76803f79b19 WatchSource:0}: Error finding container a4e557b00b24557e3a5af866ac437338816202f06f3d2832eff4b76803f79b19: Status 404 returned error can't find the container with id a4e557b00b24557e3a5af866ac437338816202f06f3d2832eff4b76803f79b19 Jul 15 13:52:07.295039 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:07.295015 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" event={"ID":"8f14cb6b-081e-4412-8b84-196fd0c45561","Type":"ContainerStarted","Data":"a4e557b00b24557e3a5af866ac437338816202f06f3d2832eff4b76803f79b19"} Jul 15 13:52:08.300780 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:08.300743 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" event={"ID":"8f14cb6b-081e-4412-8b84-196fd0c45561","Type":"ContainerStarted","Data":"c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05"} Jul 15 13:52:08.317619 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:08.317564 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" podStartSLOduration=2.31754595 podStartE2EDuration="2.31754595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:52:08.317334536 +0000 UTC m=+2743.487280889" watchObservedRunningTime="2026-07-15 13:52:08.31754595 +0000 UTC m=+2743.487492305" Jul 15 13:52:16.589485 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.589445 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" podUID="60db327b-e4f3-46cc-a009-08ea6e2e1046" containerName="main" containerID="cri-o://0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814" gracePeriod=45 Jul 15 13:52:16.851507 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.851457 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:52:16.961857 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.961832 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/60db327b-e4f3-46cc-a009-08ea6e2e1046-kube-api-access-6rqjx\") pod \"60db327b-e4f3-46cc-a009-08ea6e2e1046\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " Jul 15 13:52:16.962017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.961901 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-model-cache\") pod \"60db327b-e4f3-46cc-a009-08ea6e2e1046\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " Jul 15 13:52:16.962017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.961944 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-tmp-dir\") pod \"60db327b-e4f3-46cc-a009-08ea6e2e1046\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " Jul 15 13:52:16.962017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.961969 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-dshm\") pod \"60db327b-e4f3-46cc-a009-08ea6e2e1046\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " Jul 15 13:52:16.962017 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.961990 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-home\") pod \"60db327b-e4f3-46cc-a009-08ea6e2e1046\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " Jul 15 13:52:16.962225 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.962043 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60db327b-e4f3-46cc-a009-08ea6e2e1046-tls-certs\") pod \"60db327b-e4f3-46cc-a009-08ea6e2e1046\" (UID: \"60db327b-e4f3-46cc-a009-08ea6e2e1046\") " Jul 15 13:52:16.962225 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.962155 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-model-cache" (OuterVolumeSpecName: "model-cache") pod "60db327b-e4f3-46cc-a009-08ea6e2e1046" (UID: "60db327b-e4f3-46cc-a009-08ea6e2e1046"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:52:16.962225 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.962183 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60db327b-e4f3-46cc-a009-08ea6e2e1046" (UID: "60db327b-e4f3-46cc-a009-08ea6e2e1046"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:52:16.962344 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.962261 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-home" (OuterVolumeSpecName: "home") pod "60db327b-e4f3-46cc-a009-08ea6e2e1046" (UID: "60db327b-e4f3-46cc-a009-08ea6e2e1046"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:52:16.962344 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.962322 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-model-cache\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:52:16.962344 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.962334 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-tmp-dir\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:52:16.962344 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.962342 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-home\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:52:16.964108 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.964067 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60db327b-e4f3-46cc-a009-08ea6e2e1046-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60db327b-e4f3-46cc-a009-08ea6e2e1046" (UID: "60db327b-e4f3-46cc-a009-08ea6e2e1046"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:52:16.964211 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.964128 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-dshm" (OuterVolumeSpecName: "dshm") pod "60db327b-e4f3-46cc-a009-08ea6e2e1046" (UID: "60db327b-e4f3-46cc-a009-08ea6e2e1046"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:52:16.964211 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:16.964149 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60db327b-e4f3-46cc-a009-08ea6e2e1046-kube-api-access-6rqjx" (OuterVolumeSpecName: "kube-api-access-6rqjx") pod "60db327b-e4f3-46cc-a009-08ea6e2e1046" (UID: "60db327b-e4f3-46cc-a009-08ea6e2e1046"). InnerVolumeSpecName "kube-api-access-6rqjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:52:17.062972 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.062944 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60db327b-e4f3-46cc-a009-08ea6e2e1046-tls-certs\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:52:17.062972 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.062970 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/60db327b-e4f3-46cc-a009-08ea6e2e1046-kube-api-access-6rqjx\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:52:17.063126 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.062982 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60db327b-e4f3-46cc-a009-08ea6e2e1046-dshm\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:52:17.101344 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.101318 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:17.101451 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.101368 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:17.113684 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.113630 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:17.342495 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.342466 2582 generic.go:356] "Generic (PLEG): container finished" podID="60db327b-e4f3-46cc-a009-08ea6e2e1046" containerID="0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814" exitCode=0 Jul 15 13:52:17.342658 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.342523 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" Jul 15 13:52:17.342658 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.342551 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" event={"ID":"60db327b-e4f3-46cc-a009-08ea6e2e1046","Type":"ContainerDied","Data":"0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814"} Jul 15 13:52:17.342658 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.342595 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c" event={"ID":"60db327b-e4f3-46cc-a009-08ea6e2e1046","Type":"ContainerDied","Data":"06cd68036543d2da10ff76f95e7cd10a56a7a9e8e24b18f13629dd25e879f343"} Jul 15 13:52:17.342658 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.342616 2582 scope.go:117] "RemoveContainer" containerID="0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814" Jul 15 13:52:17.352302 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.352288 2582 scope.go:117] "RemoveContainer" containerID="0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814" Jul 15 13:52:17.352529 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:52:17.352510 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814\": container with ID starting with 0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814 not found: ID does not exist" containerID="0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814" Jul 15 13:52:17.352590 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.352536 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814"} err="failed to get container status \"0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814\": rpc error: code = NotFound desc = could not find container \"0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814\": container with ID starting with 0af4cb2ac6c602f9811858e4f84bd94d759810bd80a038009bdde821de822814 not found: ID does not exist" Jul 15 13:52:17.355435 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.355416 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:52:17.362704 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.362685 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c"] Jul 15 13:52:17.366238 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.366190 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/auth-test-model-7abf62-kserve-64d95fd8d7-n5s8c"] Jul 15 13:52:17.460668 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:17.460643 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60db327b-e4f3-46cc-a009-08ea6e2e1046" path="/var/lib/kubelet/pods/60db327b-e4f3-46cc-a009-08ea6e2e1046/volumes" Jul 15 13:52:21.785581 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:21.785549 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:52:44.020353 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:44.020319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:52:47.445055 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:47.444967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:52:48.885129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:48.885093 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:52:59.112785 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:52:59.112752 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:53:00.088981 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:00.088938 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:53:00.191010 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:00.190978 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:53:20.695361 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:20.695326 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:53:39.070367 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:39.070330 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p"] Jul 15 13:53:39.096082 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:39.096058 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:53:39.825021 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:39.824989 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:53:41.188628 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:41.188583 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:53:47.593561 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.593521 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt"] Jul 15 13:53:47.594046 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.593856 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60db327b-e4f3-46cc-a009-08ea6e2e1046" containerName="main" Jul 15 13:53:47.594046 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.593868 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60db327b-e4f3-46cc-a009-08ea6e2e1046" containerName="main" Jul 15 13:53:47.594046 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.593939 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="60db327b-e4f3-46cc-a009-08ea6e2e1046" containerName="main" Jul 15 13:53:47.596594 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.596563 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.598443 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.598427 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7abf62\"/\"rate-test-model-7abf62-kserve-self-signed-certs\"" Jul 15 13:53:47.608645 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.608624 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt"] Jul 15 13:53:47.639968 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.639942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:53:47.711397 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.711373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-home\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.711503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.711419 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-model-cache\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.711503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.711436 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-dshm\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.711503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.711455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcf6\" (UniqueName: \"kubernetes.io/projected/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-kube-api-access-mkcf6\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.711503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.711473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tmp-dir\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.711503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.711503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tls-certs\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.812942 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.812915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-home\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.813091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.812968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-model-cache\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.813091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.812988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-dshm\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.813091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.813004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcf6\" (UniqueName: \"kubernetes.io/projected/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-kube-api-access-mkcf6\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.813091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.813022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tmp-dir\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.813091 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.813056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tls-certs\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.813366 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.813348 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-home\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.813454 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.813424 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tmp-dir\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.813570 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.813483 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-model-cache\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.815170 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.815149 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-dshm\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.815469 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.815453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tls-certs\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.820955 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.820934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcf6\" (UniqueName: \"kubernetes.io/projected/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-kube-api-access-mkcf6\") pod \"rate-test-model-7abf62-kserve-6b4866c555-xh9dt\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:47.906706 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:47.906652 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:48.037340 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:48.037315 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt"] Jul 15 13:53:48.039389 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:53:48.039363 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7f8eb64_c9fb_4d30_9f12_1d03795f4251.slice/crio-789c705ceacd2f062e59084910d62ab0a8e0773b5590337185cb9c7a692515a8 WatchSource:0}: Error finding container 789c705ceacd2f062e59084910d62ab0a8e0773b5590337185cb9c7a692515a8: Status 404 returned error can't find the container with id 789c705ceacd2f062e59084910d62ab0a8e0773b5590337185cb9c7a692515a8 Jul 15 13:53:48.758189 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:48.758151 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" event={"ID":"e7f8eb64-c9fb-4d30-9f12-1d03795f4251","Type":"ContainerStarted","Data":"a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec"} Jul 15 13:53:48.758583 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:48.758201 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" event={"ID":"e7f8eb64-c9fb-4d30-9f12-1d03795f4251","Type":"ContainerStarted","Data":"789c705ceacd2f062e59084910d62ab0a8e0773b5590337185cb9c7a692515a8"} Jul 15 13:53:48.779375 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:48.779327 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" podStartSLOduration=1.7793034909999998 podStartE2EDuration="1.779303491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:53:48.777096472 +0000 UTC m=+2843.947042848" watchObservedRunningTime="2026-07-15 13:53:48.779303491 +0000 UTC m=+2843.949249902" Jul 15 13:53:54.092115 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.092077 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" podUID="8f14cb6b-081e-4412-8b84-196fd0c45561" containerName="main" containerID="cri-o://c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05" gracePeriod=45 Jul 15 13:53:54.344910 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.344854 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:53:54.465941 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.465914 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-dshm\") pod \"8f14cb6b-081e-4412-8b84-196fd0c45561\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " Jul 15 13:53:54.466082 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.465954 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-tmp-dir\") pod \"8f14cb6b-081e-4412-8b84-196fd0c45561\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " Jul 15 13:53:54.466082 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.465979 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvgj4\" (UniqueName: \"kubernetes.io/projected/8f14cb6b-081e-4412-8b84-196fd0c45561-kube-api-access-fvgj4\") pod \"8f14cb6b-081e-4412-8b84-196fd0c45561\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " Jul 15 13:53:54.466082 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.465998 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-home\") pod \"8f14cb6b-081e-4412-8b84-196fd0c45561\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " Jul 15 13:53:54.466245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.466149 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f14cb6b-081e-4412-8b84-196fd0c45561-tls-certs\") pod \"8f14cb6b-081e-4412-8b84-196fd0c45561\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " Jul 15 13:53:54.466245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.466189 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f14cb6b-081e-4412-8b84-196fd0c45561" (UID: "8f14cb6b-081e-4412-8b84-196fd0c45561"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:53:54.466245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.466210 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-model-cache\") pod \"8f14cb6b-081e-4412-8b84-196fd0c45561\" (UID: \"8f14cb6b-081e-4412-8b84-196fd0c45561\") " Jul 15 13:53:54.466245 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.466232 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-home" (OuterVolumeSpecName: "home") pod "8f14cb6b-081e-4412-8b84-196fd0c45561" (UID: "8f14cb6b-081e-4412-8b84-196fd0c45561"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:53:54.466439 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.466360 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f14cb6b-081e-4412-8b84-196fd0c45561" (UID: "8f14cb6b-081e-4412-8b84-196fd0c45561"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:53:54.466511 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.466495 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-home\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:53:54.466571 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.466513 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-model-cache\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:53:54.466571 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.466523 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-tmp-dir\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:53:54.467895 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.467875 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-dshm" (OuterVolumeSpecName: "dshm") pod "8f14cb6b-081e-4412-8b84-196fd0c45561" (UID: "8f14cb6b-081e-4412-8b84-196fd0c45561"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:53:54.468031 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.468013 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f14cb6b-081e-4412-8b84-196fd0c45561-kube-api-access-fvgj4" (OuterVolumeSpecName: "kube-api-access-fvgj4") pod "8f14cb6b-081e-4412-8b84-196fd0c45561" (UID: "8f14cb6b-081e-4412-8b84-196fd0c45561"). InnerVolumeSpecName "kube-api-access-fvgj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:53:54.468115 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.468099 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f14cb6b-081e-4412-8b84-196fd0c45561-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f14cb6b-081e-4412-8b84-196fd0c45561" (UID: "8f14cb6b-081e-4412-8b84-196fd0c45561"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:53:54.567876 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.567849 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f14cb6b-081e-4412-8b84-196fd0c45561-dshm\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:53:54.567876 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.567873 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvgj4\" (UniqueName: \"kubernetes.io/projected/8f14cb6b-081e-4412-8b84-196fd0c45561-kube-api-access-fvgj4\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:53:54.568020 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.567887 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f14cb6b-081e-4412-8b84-196fd0c45561-tls-certs\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:53:54.789479 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.789447 2582 generic.go:356] "Generic (PLEG): container finished" podID="8f14cb6b-081e-4412-8b84-196fd0c45561" containerID="c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05" exitCode=0 Jul 15 13:53:54.789678 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.789495 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" Jul 15 13:53:54.789678 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.789532 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" event={"ID":"8f14cb6b-081e-4412-8b84-196fd0c45561","Type":"ContainerDied","Data":"c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05"} Jul 15 13:53:54.789678 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.789575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p" event={"ID":"8f14cb6b-081e-4412-8b84-196fd0c45561","Type":"ContainerDied","Data":"a4e557b00b24557e3a5af866ac437338816202f06f3d2832eff4b76803f79b19"} Jul 15 13:53:54.789678 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.789592 2582 scope.go:117] "RemoveContainer" containerID="c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05" Jul 15 13:53:54.798239 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.798225 2582 scope.go:117] "RemoveContainer" containerID="c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05" Jul 15 13:53:54.798485 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:53:54.798467 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05\": container with ID starting with c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05 not found: ID does not exist" containerID="c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05" Jul 15 13:53:54.798523 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.798493 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05"} err="failed to get container status \"c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05\": rpc error: code = NotFound desc = could not find container \"c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05\": container with ID starting with c6ae6c5d7181ae97c59055b81d0965688e3d5276a87fa19de42b62085cd2aa05 not found: ID does not exist" Jul 15 13:53:54.808379 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.808357 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p"] Jul 15 13:53:54.809865 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:54.809849 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/sub-test-model-7abf62-kserve-5bb89998f9-4zw8p"] Jul 15 13:53:55.461631 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:55.461599 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f14cb6b-081e-4412-8b84-196fd0c45561" path="/var/lib/kubelet/pods/8f14cb6b-081e-4412-8b84-196fd0c45561/volumes" Jul 15 13:53:57.906970 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:57.906928 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:57.907384 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:57.907000 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:57.919378 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:57.919349 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:53:58.820271 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:53:58.820245 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:54:02.797882 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:02.797848 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:54:19.511601 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:19.511541 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:54:20.202755 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:20.202706 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:54:27.380032 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:27.379995 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:54:28.533960 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:28.533927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:54:28.635148 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:28.635104 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:54:30.853549 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:30.853505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:54:34.887229 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:34.887195 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:54:49.692936 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:54:49.692901 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:55:06.192484 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:06.192447 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:55:13.644796 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:13.644764 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt"] Jul 15 13:55:13.669999 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:13.669968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:14.642505 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:14.642457 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:15.643736 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.643676 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-140115524557360"] Jul 15 13:55:15.644098 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.644009 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f14cb6b-081e-4412-8b84-196fd0c45561" containerName="main" Jul 15 13:55:15.644098 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.644020 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f14cb6b-081e-4412-8b84-196fd0c45561" containerName="main" Jul 15 13:55:15.644098 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.644078 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f14cb6b-081e-4412-8b84-196fd0c45561" containerName="main" Jul 15 13:55:15.646652 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.646636 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524557360" Jul 15 13:55:15.648607 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.648583 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8zcqs\"" Jul 15 13:55:15.651928 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.651907 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524557360"] Jul 15 13:55:15.697499 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.697472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbs5h\" (UniqueName: \"kubernetes.io/projected/14c64bb4-e7ea-4f73-9ba7-5f095e98d845-kube-api-access-qbs5h\") pod \"test-curl-7422-140115524557360\" (UID: \"14c64bb4-e7ea-4f73-9ba7-5f095e98d845\") " pod="opendatahub/test-curl-7422-140115524557360" Jul 15 13:55:15.798574 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.798552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbs5h\" (UniqueName: \"kubernetes.io/projected/14c64bb4-e7ea-4f73-9ba7-5f095e98d845-kube-api-access-qbs5h\") pod \"test-curl-7422-140115524557360\" (UID: \"14c64bb4-e7ea-4f73-9ba7-5f095e98d845\") " pod="opendatahub/test-curl-7422-140115524557360" Jul 15 13:55:15.806420 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.806395 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbs5h\" (UniqueName: \"kubernetes.io/projected/14c64bb4-e7ea-4f73-9ba7-5f095e98d845-kube-api-access-qbs5h\") pod \"test-curl-7422-140115524557360\" (UID: \"14c64bb4-e7ea-4f73-9ba7-5f095e98d845\") " pod="opendatahub/test-curl-7422-140115524557360" Jul 15 13:55:15.956677 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:15.956654 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524557360" Jul 15 13:55:16.086318 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:16.086287 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524557360"] Jul 15 13:55:16.089650 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:55:16.089623 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14c64bb4_e7ea_4f73_9ba7_5f095e98d845.slice/crio-71df39c9a184be60b014810345a1eeabde77c4a11aa6148b762b0a12149cbed2 WatchSource:0}: Error finding container 71df39c9a184be60b014810345a1eeabde77c4a11aa6148b762b0a12149cbed2: Status 404 returned error can't find the container with id 71df39c9a184be60b014810345a1eeabde77c4a11aa6148b762b0a12149cbed2 Jul 15 13:55:16.091280 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:16.091262 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:55:16.168384 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:16.168353 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524557360" event={"ID":"14c64bb4-e7ea-4f73-9ba7-5f095e98d845","Type":"ContainerStarted","Data":"71df39c9a184be60b014810345a1eeabde77c4a11aa6148b762b0a12149cbed2"} Jul 15 13:55:17.173877 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:17.173797 2582 generic.go:356] "Generic (PLEG): container finished" podID="14c64bb4-e7ea-4f73-9ba7-5f095e98d845" containerID="8b747f276cc0ef985f1080940623e59b5d8a2fcf9a21f6bd0641148e38fc3f4f" exitCode=0 Jul 15 13:55:17.174221 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:17.173882 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524557360" event={"ID":"14c64bb4-e7ea-4f73-9ba7-5f095e98d845","Type":"ContainerDied","Data":"8b747f276cc0ef985f1080940623e59b5d8a2fcf9a21f6bd0641148e38fc3f4f"} Jul 15 13:55:18.309909 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.309887 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524557360" Jul 15 13:55:18.328380 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.328355 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7422-140115524557360_14c64bb4-e7ea-4f73-9ba7-5f095e98d845/test-curl-7422-140115524557360/0.log" Jul 15 13:55:18.350650 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.350624 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-140115524557360"] Jul 15 13:55:18.353124 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.353104 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-140115524557360"] Jul 15 13:55:18.420006 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.419981 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbs5h\" (UniqueName: \"kubernetes.io/projected/14c64bb4-e7ea-4f73-9ba7-5f095e98d845-kube-api-access-qbs5h\") pod \"14c64bb4-e7ea-4f73-9ba7-5f095e98d845\" (UID: \"14c64bb4-e7ea-4f73-9ba7-5f095e98d845\") " Jul 15 13:55:18.422095 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.422062 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c64bb4-e7ea-4f73-9ba7-5f095e98d845-kube-api-access-qbs5h" (OuterVolumeSpecName: "kube-api-access-qbs5h") pod "14c64bb4-e7ea-4f73-9ba7-5f095e98d845" (UID: "14c64bb4-e7ea-4f73-9ba7-5f095e98d845"). InnerVolumeSpecName "kube-api-access-qbs5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:55:18.475567 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.475541 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-140115524543024"] Jul 15 13:55:18.475867 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.475854 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c64bb4-e7ea-4f73-9ba7-5f095e98d845" containerName="test-curl-7422-140115524557360" Jul 15 13:55:18.475924 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.475869 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c64bb4-e7ea-4f73-9ba7-5f095e98d845" containerName="test-curl-7422-140115524557360" Jul 15 13:55:18.475960 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.475933 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c64bb4-e7ea-4f73-9ba7-5f095e98d845" containerName="test-curl-7422-140115524557360" Jul 15 13:55:18.478768 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.478753 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524543024" Jul 15 13:55:18.485894 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.485874 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524543024"] Jul 15 13:55:18.520853 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.520829 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbs5h\" (UniqueName: \"kubernetes.io/projected/14c64bb4-e7ea-4f73-9ba7-5f095e98d845-kube-api-access-qbs5h\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:18.621440 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.621414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28kc\" (UniqueName: \"kubernetes.io/projected/6126ead5-fba4-4d98-9456-28f1dd8cd41c-kube-api-access-n28kc\") pod \"test-curl-7422-140115524543024\" (UID: \"6126ead5-fba4-4d98-9456-28f1dd8cd41c\") " pod="opendatahub/test-curl-7422-140115524543024" Jul 15 13:55:18.722151 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.722131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28kc\" (UniqueName: \"kubernetes.io/projected/6126ead5-fba4-4d98-9456-28f1dd8cd41c-kube-api-access-n28kc\") pod \"test-curl-7422-140115524543024\" (UID: \"6126ead5-fba4-4d98-9456-28f1dd8cd41c\") " pod="opendatahub/test-curl-7422-140115524543024" Jul 15 13:55:18.730071 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.730015 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28kc\" (UniqueName: \"kubernetes.io/projected/6126ead5-fba4-4d98-9456-28f1dd8cd41c-kube-api-access-n28kc\") pod \"test-curl-7422-140115524543024\" (UID: \"6126ead5-fba4-4d98-9456-28f1dd8cd41c\") " pod="opendatahub/test-curl-7422-140115524543024" Jul 15 13:55:18.788273 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.788242 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524543024" Jul 15 13:55:18.913617 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:18.913584 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524543024"] Jul 15 13:55:18.915874 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:55:18.915847 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6126ead5_fba4_4d98_9456_28f1dd8cd41c.slice/crio-66b1ac29c419c760ffc4fd18b629f5b96309b44c07119def2985734b87b6fdeb WatchSource:0}: Error finding container 66b1ac29c419c760ffc4fd18b629f5b96309b44c07119def2985734b87b6fdeb: Status 404 returned error can't find the container with id 66b1ac29c419c760ffc4fd18b629f5b96309b44c07119def2985734b87b6fdeb Jul 15 13:55:19.183496 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:19.183423 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524543024" event={"ID":"6126ead5-fba4-4d98-9456-28f1dd8cd41c","Type":"ContainerStarted","Data":"66b1ac29c419c760ffc4fd18b629f5b96309b44c07119def2985734b87b6fdeb"} Jul 15 13:55:19.184484 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:19.184467 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524557360" Jul 15 13:55:19.184597 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:19.184466 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71df39c9a184be60b014810345a1eeabde77c4a11aa6148b762b0a12149cbed2" Jul 15 13:55:19.194557 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:55:19.194527 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7422-140115524557360\" is forbidden: User \"system:node:ip-10-0-131-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-140.ec2.internal' and this object" podUID="14c64bb4-e7ea-4f73-9ba7-5f095e98d845" pod="opendatahub/test-curl-7422-140115524557360" Jul 15 13:55:19.460744 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:19.460701 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c64bb4-e7ea-4f73-9ba7-5f095e98d845" path="/var/lib/kubelet/pods/14c64bb4-e7ea-4f73-9ba7-5f095e98d845/volumes" Jul 15 13:55:20.190178 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:20.190142 2582 generic.go:356] "Generic (PLEG): container finished" podID="6126ead5-fba4-4d98-9456-28f1dd8cd41c" containerID="0850c313b63dcb8df1bb5412de616e5292e48d2db70ac545382a1b44b0a9712f" exitCode=0 Jul 15 13:55:20.190339 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:20.190209 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524543024" event={"ID":"6126ead5-fba4-4d98-9456-28f1dd8cd41c","Type":"ContainerDied","Data":"0850c313b63dcb8df1bb5412de616e5292e48d2db70ac545382a1b44b0a9712f"} Jul 15 13:55:20.591202 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:20.591171 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:55:21.340570 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.340543 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524543024" Jul 15 13:55:21.360072 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.360049 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7422-140115524543024_6126ead5-fba4-4d98-9456-28f1dd8cd41c/test-curl-7422-140115524543024/0.log" Jul 15 13:55:21.384675 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.384648 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-140115524543024"] Jul 15 13:55:21.388051 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.388026 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-140115524543024"] Jul 15 13:55:21.441916 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.441896 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n28kc\" (UniqueName: \"kubernetes.io/projected/6126ead5-fba4-4d98-9456-28f1dd8cd41c-kube-api-access-n28kc\") pod \"6126ead5-fba4-4d98-9456-28f1dd8cd41c\" (UID: \"6126ead5-fba4-4d98-9456-28f1dd8cd41c\") " Jul 15 13:55:21.443854 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.443829 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6126ead5-fba4-4d98-9456-28f1dd8cd41c-kube-api-access-n28kc" (OuterVolumeSpecName: "kube-api-access-n28kc") pod "6126ead5-fba4-4d98-9456-28f1dd8cd41c" (UID: "6126ead5-fba4-4d98-9456-28f1dd8cd41c"). InnerVolumeSpecName "kube-api-access-n28kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:55:21.460577 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.460548 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6126ead5-fba4-4d98-9456-28f1dd8cd41c" path="/var/lib/kubelet/pods/6126ead5-fba4-4d98-9456-28f1dd8cd41c/volumes" Jul 15 13:55:21.543019 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.542996 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n28kc\" (UniqueName: \"kubernetes.io/projected/6126ead5-fba4-4d98-9456-28f1dd8cd41c-kube-api-access-n28kc\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:21.560059 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.560038 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:21.560404 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.560391 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6126ead5-fba4-4d98-9456-28f1dd8cd41c" containerName="test-curl-7422-140115524543024" Jul 15 13:55:21.560452 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.560406 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6126ead5-fba4-4d98-9456-28f1dd8cd41c" containerName="test-curl-7422-140115524543024" Jul 15 13:55:21.560488 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.560469 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6126ead5-fba4-4d98-9456-28f1dd8cd41c" containerName="test-curl-7422-140115524543024" Jul 15 13:55:21.563437 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.563418 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:21.568512 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.568486 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:21.644457 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.644427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2tp\" (UniqueName: \"kubernetes.io/projected/bccca512-673d-429a-bed9-060f73cb5100-kube-api-access-2h2tp\") pod \"test-curl-7422-140115524522608\" (UID: \"bccca512-673d-429a-bed9-060f73cb5100\") " pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:21.745852 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.745827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2tp\" (UniqueName: \"kubernetes.io/projected/bccca512-673d-429a-bed9-060f73cb5100-kube-api-access-2h2tp\") pod \"test-curl-7422-140115524522608\" (UID: \"bccca512-673d-429a-bed9-060f73cb5100\") " pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:21.753482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.753452 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2tp\" (UniqueName: \"kubernetes.io/projected/bccca512-673d-429a-bed9-060f73cb5100-kube-api-access-2h2tp\") pod \"test-curl-7422-140115524522608\" (UID: \"bccca512-673d-429a-bed9-060f73cb5100\") " pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:21.875346 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.875313 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:21.998921 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:21.998897 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:22.001218 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:55:22.001189 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbccca512_673d_429a_bed9_060f73cb5100.slice/crio-0d72b07c84d84a1efa4591883f411ec58c581d65cde03faae4448646feb3ca97 WatchSource:0}: Error finding container 0d72b07c84d84a1efa4591883f411ec58c581d65cde03faae4448646feb3ca97: Status 404 returned error can't find the container with id 0d72b07c84d84a1efa4591883f411ec58c581d65cde03faae4448646feb3ca97 Jul 15 13:55:22.199950 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:22.199921 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524522608" event={"ID":"bccca512-673d-429a-bed9-060f73cb5100","Type":"ContainerStarted","Data":"0d72b07c84d84a1efa4591883f411ec58c581d65cde03faae4448646feb3ca97"} Jul 15 13:55:22.201009 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:22.200993 2582 scope.go:117] "RemoveContainer" containerID="0850c313b63dcb8df1bb5412de616e5292e48d2db70ac545382a1b44b0a9712f" Jul 15 13:55:22.201009 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:22.201007 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524543024" Jul 15 13:55:23.207000 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:23.206964 2582 generic.go:356] "Generic (PLEG): container finished" podID="bccca512-673d-429a-bed9-060f73cb5100" containerID="411d18bcacc40eacd42f52566e6690b6de895b1a40507bbfbe739ed5ac999935" exitCode=0 Jul 15 13:55:23.207429 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:23.207046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524522608" event={"ID":"bccca512-673d-429a-bed9-060f73cb5100","Type":"ContainerDied","Data":"411d18bcacc40eacd42f52566e6690b6de895b1a40507bbfbe739ed5ac999935"} Jul 15 13:55:24.346475 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.346445 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:24.365407 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.365383 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7422-140115524522608_bccca512-673d-429a-bed9-060f73cb5100/test-curl-7422-140115524522608/0.log" Jul 15 13:55:24.386671 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.386623 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:24.392856 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.392835 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:24.466375 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.466318 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h2tp\" (UniqueName: \"kubernetes.io/projected/bccca512-673d-429a-bed9-060f73cb5100-kube-api-access-2h2tp\") pod \"bccca512-673d-429a-bed9-060f73cb5100\" (UID: \"bccca512-673d-429a-bed9-060f73cb5100\") " Jul 15 13:55:24.468264 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.468244 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccca512-673d-429a-bed9-060f73cb5100-kube-api-access-2h2tp" (OuterVolumeSpecName: "kube-api-access-2h2tp") pod "bccca512-673d-429a-bed9-060f73cb5100" (UID: "bccca512-673d-429a-bed9-060f73cb5100"). InnerVolumeSpecName "kube-api-access-2h2tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:55:24.511499 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.511477 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:24.511814 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.511800 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccca512-673d-429a-bed9-060f73cb5100" containerName="test-curl-7422-140115524522608" Jul 15 13:55:24.511862 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.511815 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccca512-673d-429a-bed9-060f73cb5100" containerName="test-curl-7422-140115524522608" Jul 15 13:55:24.511899 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.511885 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccca512-673d-429a-bed9-060f73cb5100" containerName="test-curl-7422-140115524522608" Jul 15 13:55:24.514623 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.514609 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:24.521209 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.521157 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:24.567710 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.567692 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2tp\" (UniqueName: \"kubernetes.io/projected/bccca512-673d-429a-bed9-060f73cb5100-kube-api-access-2h2tp\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:24.669026 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.669000 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fvc\" (UniqueName: \"kubernetes.io/projected/67904ef6-f5e3-496f-b0b3-06caafe9ddbc-kube-api-access-h7fvc\") pod \"test-curl-7422-140115524522608\" (UID: \"67904ef6-f5e3-496f-b0b3-06caafe9ddbc\") " pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:24.770056 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.769986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fvc\" (UniqueName: \"kubernetes.io/projected/67904ef6-f5e3-496f-b0b3-06caafe9ddbc-kube-api-access-h7fvc\") pod \"test-curl-7422-140115524522608\" (UID: \"67904ef6-f5e3-496f-b0b3-06caafe9ddbc\") " pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:24.777751 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.777710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fvc\" (UniqueName: \"kubernetes.io/projected/67904ef6-f5e3-496f-b0b3-06caafe9ddbc-kube-api-access-h7fvc\") pod \"test-curl-7422-140115524522608\" (UID: \"67904ef6-f5e3-496f-b0b3-06caafe9ddbc\") " pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:24.823694 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.823671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:24.949538 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:24.949516 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:24.952723 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:55:24.952686 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67904ef6_f5e3_496f_b0b3_06caafe9ddbc.slice/crio-884b891ab8a7ada3f17ae63e954908190fbc18a12bb83f7e2b8489b374f72f9c WatchSource:0}: Error finding container 884b891ab8a7ada3f17ae63e954908190fbc18a12bb83f7e2b8489b374f72f9c: Status 404 returned error can't find the container with id 884b891ab8a7ada3f17ae63e954908190fbc18a12bb83f7e2b8489b374f72f9c Jul 15 13:55:25.218178 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:25.218153 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:25.218327 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:25.218183 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d72b07c84d84a1efa4591883f411ec58c581d65cde03faae4448646feb3ca97" Jul 15 13:55:25.219232 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:25.219212 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524522608" event={"ID":"67904ef6-f5e3-496f-b0b3-06caafe9ddbc","Type":"ContainerStarted","Data":"884b891ab8a7ada3f17ae63e954908190fbc18a12bb83f7e2b8489b374f72f9c"} Jul 15 13:55:25.229075 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:25.229045 2582 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7422-140115524522608" oldPodUID="bccca512-673d-429a-bed9-060f73cb5100" podUID="67904ef6-f5e3-496f-b0b3-06caafe9ddbc" Jul 15 13:55:25.465782 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:25.465751 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bccca512-673d-429a-bed9-060f73cb5100" path="/var/lib/kubelet/pods/bccca512-673d-429a-bed9-060f73cb5100/volumes" Jul 15 13:55:26.225639 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:26.225608 2582 generic.go:356] "Generic (PLEG): container finished" podID="67904ef6-f5e3-496f-b0b3-06caafe9ddbc" containerID="67a3ec65dc943bdbb1af09b3f3ff5a297679dc24d243dcc30b3d7c3fed4eb2b6" exitCode=0 Jul 15 13:55:26.225901 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:26.225694 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524522608" event={"ID":"67904ef6-f5e3-496f-b0b3-06caafe9ddbc","Type":"ContainerDied","Data":"67a3ec65dc943bdbb1af09b3f3ff5a297679dc24d243dcc30b3d7c3fed4eb2b6"} Jul 15 13:55:27.377172 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.377149 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:27.396663 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.396615 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7422-140115524522608_67904ef6-f5e3-496f-b0b3-06caafe9ddbc/test-curl-7422-140115524522608/0.log" Jul 15 13:55:27.418460 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.418385 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:27.420003 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.419983 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-140115524522608"] Jul 15 13:55:27.492677 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.492611 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7fvc\" (UniqueName: \"kubernetes.io/projected/67904ef6-f5e3-496f-b0b3-06caafe9ddbc-kube-api-access-h7fvc\") pod \"67904ef6-f5e3-496f-b0b3-06caafe9ddbc\" (UID: \"67904ef6-f5e3-496f-b0b3-06caafe9ddbc\") " Jul 15 13:55:27.494632 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.494609 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67904ef6-f5e3-496f-b0b3-06caafe9ddbc-kube-api-access-h7fvc" (OuterVolumeSpecName: "kube-api-access-h7fvc") pod "67904ef6-f5e3-496f-b0b3-06caafe9ddbc" (UID: "67904ef6-f5e3-496f-b0b3-06caafe9ddbc"). InnerVolumeSpecName "kube-api-access-h7fvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:55:27.546436 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.546409 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:27.546766 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.546750 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67904ef6-f5e3-496f-b0b3-06caafe9ddbc" containerName="test-curl-7422-140115524522608" Jul 15 13:55:27.546831 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.546769 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="67904ef6-f5e3-496f-b0b3-06caafe9ddbc" containerName="test-curl-7422-140115524522608" Jul 15 13:55:27.546874 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.546865 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="67904ef6-f5e3-496f-b0b3-06caafe9ddbc" containerName="test-curl-7422-140115524522608" Jul 15 13:55:27.549692 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.549673 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:27.554560 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.554541 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:27.593647 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.593625 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7fvc\" (UniqueName: \"kubernetes.io/projected/67904ef6-f5e3-496f-b0b3-06caafe9ddbc-kube-api-access-h7fvc\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:27.694929 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.694902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9hxr\" (UniqueName: \"kubernetes.io/projected/635134b8-0556-4ded-81d4-9df9f079acbb-kube-api-access-g9hxr\") pod \"test-curl-7422-140115524525632\" (UID: \"635134b8-0556-4ded-81d4-9df9f079acbb\") " pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:27.796370 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.796264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9hxr\" (UniqueName: \"kubernetes.io/projected/635134b8-0556-4ded-81d4-9df9f079acbb-kube-api-access-g9hxr\") pod \"test-curl-7422-140115524525632\" (UID: \"635134b8-0556-4ded-81d4-9df9f079acbb\") " pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:27.804210 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.804183 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9hxr\" (UniqueName: \"kubernetes.io/projected/635134b8-0556-4ded-81d4-9df9f079acbb-kube-api-access-g9hxr\") pod \"test-curl-7422-140115524525632\" (UID: \"635134b8-0556-4ded-81d4-9df9f079acbb\") " pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:27.859660 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.859623 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:27.987502 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:27.987474 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:27.990065 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:55:27.990041 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod635134b8_0556_4ded_81d4_9df9f079acbb.slice/crio-bfd50f558b12b26de64e2d75c924513095c2a96a5b08813c1946468046586705 WatchSource:0}: Error finding container bfd50f558b12b26de64e2d75c924513095c2a96a5b08813c1946468046586705: Status 404 returned error can't find the container with id bfd50f558b12b26de64e2d75c924513095c2a96a5b08813c1946468046586705 Jul 15 13:55:28.237322 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:28.237290 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524525632" event={"ID":"635134b8-0556-4ded-81d4-9df9f079acbb","Type":"ContainerStarted","Data":"bfd50f558b12b26de64e2d75c924513095c2a96a5b08813c1946468046586705"} Jul 15 13:55:28.238553 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:28.238533 2582 scope.go:117] "RemoveContainer" containerID="67a3ec65dc943bdbb1af09b3f3ff5a297679dc24d243dcc30b3d7c3fed4eb2b6" Jul 15 13:55:28.238553 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:28.238544 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524522608" Jul 15 13:55:28.666045 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:28.665961 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" podUID="e7f8eb64-c9fb-4d30-9f12-1d03795f4251" containerName="main" containerID="cri-o://a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec" gracePeriod=45 Jul 15 13:55:28.809927 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:28.809892 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" podUID="e7f8eb64-c9fb-4d30-9f12-1d03795f4251" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": dial tcp 10.132.0.42:8000: connect: connection refused" Jul 15 13:55:28.945694 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:28.945671 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:55:29.109375 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109348 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-model-cache\") pod \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " Jul 15 13:55:29.109502 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109385 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkcf6\" (UniqueName: \"kubernetes.io/projected/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-kube-api-access-mkcf6\") pod \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " Jul 15 13:55:29.109502 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109448 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-home\") pod \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " Jul 15 13:55:29.109502 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109478 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tls-certs\") pod \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " Jul 15 13:55:29.109674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109512 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tmp-dir\") pod \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " Jul 15 13:55:29.109674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109578 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-dshm\") pod \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\" (UID: \"e7f8eb64-c9fb-4d30-9f12-1d03795f4251\") " Jul 15 13:55:29.109674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109638 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7f8eb64-c9fb-4d30-9f12-1d03795f4251" (UID: "e7f8eb64-c9fb-4d30-9f12-1d03795f4251"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:55:29.109674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109653 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-home" (OuterVolumeSpecName: "home") pod "e7f8eb64-c9fb-4d30-9f12-1d03795f4251" (UID: "e7f8eb64-c9fb-4d30-9f12-1d03795f4251"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:55:29.109911 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109831 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7f8eb64-c9fb-4d30-9f12-1d03795f4251" (UID: "e7f8eb64-c9fb-4d30-9f12-1d03795f4251"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:55:29.109911 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109850 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-model-cache\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:29.109911 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.109863 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-home\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:29.111510 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.111490 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-dshm" (OuterVolumeSpecName: "dshm") pod "e7f8eb64-c9fb-4d30-9f12-1d03795f4251" (UID: "e7f8eb64-c9fb-4d30-9f12-1d03795f4251"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:55:29.111792 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.111520 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-kube-api-access-mkcf6" (OuterVolumeSpecName: "kube-api-access-mkcf6") pod "e7f8eb64-c9fb-4d30-9f12-1d03795f4251" (UID: "e7f8eb64-c9fb-4d30-9f12-1d03795f4251"). InnerVolumeSpecName "kube-api-access-mkcf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:55:29.111792 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.111552 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7f8eb64-c9fb-4d30-9f12-1d03795f4251" (UID: "e7f8eb64-c9fb-4d30-9f12-1d03795f4251"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:55:29.210888 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.210866 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkcf6\" (UniqueName: \"kubernetes.io/projected/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-kube-api-access-mkcf6\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:29.210976 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.210888 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tls-certs\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:29.210976 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.210898 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-tmp-dir\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:29.210976 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.210906 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f8eb64-c9fb-4d30-9f12-1d03795f4251-dshm\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:29.244846 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.244818 2582 generic.go:356] "Generic (PLEG): container finished" podID="635134b8-0556-4ded-81d4-9df9f079acbb" containerID="fa2f336f914e6081f18fc20e8d8873b6587541ab31d3672f0768216489d75d53" exitCode=0 Jul 15 13:55:29.244958 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.244898 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524525632" event={"ID":"635134b8-0556-4ded-81d4-9df9f079acbb","Type":"ContainerDied","Data":"fa2f336f914e6081f18fc20e8d8873b6587541ab31d3672f0768216489d75d53"} Jul 15 13:55:29.246016 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.245999 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7f8eb64-c9fb-4d30-9f12-1d03795f4251" containerID="a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec" exitCode=0 Jul 15 13:55:29.246105 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.246041 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" Jul 15 13:55:29.246105 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.246067 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" event={"ID":"e7f8eb64-c9fb-4d30-9f12-1d03795f4251","Type":"ContainerDied","Data":"a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec"} Jul 15 13:55:29.246105 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.246094 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt" event={"ID":"e7f8eb64-c9fb-4d30-9f12-1d03795f4251","Type":"ContainerDied","Data":"789c705ceacd2f062e59084910d62ab0a8e0773b5590337185cb9c7a692515a8"} Jul 15 13:55:29.246269 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.246116 2582 scope.go:117] "RemoveContainer" containerID="a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec" Jul 15 13:55:29.254247 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.254233 2582 scope.go:117] "RemoveContainer" containerID="a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec" Jul 15 13:55:29.254507 ip-10-0-131-140 kubenswrapper[2582]: E0715 13:55:29.254490 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec\": container with ID starting with a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec not found: ID does not exist" containerID="a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec" Jul 15 13:55:29.254557 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.254517 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec"} err="failed to get container status \"a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec\": rpc error: code = NotFound desc = could not find container \"a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec\": container with ID starting with a4ae94c41cb83ca39697a89792568604a4bdf9ee6ae017ac3b97fcaa26e6abec not found: ID does not exist" Jul 15 13:55:29.268899 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.268874 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt"] Jul 15 13:55:29.270358 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.270338 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7abf62/rate-test-model-7abf62-kserve-6b4866c555-xh9dt"] Jul 15 13:55:29.461279 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.461221 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67904ef6-f5e3-496f-b0b3-06caafe9ddbc" path="/var/lib/kubelet/pods/67904ef6-f5e3-496f-b0b3-06caafe9ddbc/volumes" Jul 15 13:55:29.461589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:29.461576 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f8eb64-c9fb-4d30-9f12-1d03795f4251" path="/var/lib/kubelet/pods/e7f8eb64-c9fb-4d30-9f12-1d03795f4251/volumes" Jul 15 13:55:30.389947 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.389924 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:30.408810 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.408791 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7422-140115524525632_635134b8-0556-4ded-81d4-9df9f079acbb/test-curl-7422-140115524525632/0.log" Jul 15 13:55:30.429060 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.429030 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:30.434463 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.434440 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:30.526992 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.526940 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9hxr\" (UniqueName: \"kubernetes.io/projected/635134b8-0556-4ded-81d4-9df9f079acbb-kube-api-access-g9hxr\") pod \"635134b8-0556-4ded-81d4-9df9f079acbb\" (UID: \"635134b8-0556-4ded-81d4-9df9f079acbb\") " Jul 15 13:55:30.528938 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.528918 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635134b8-0556-4ded-81d4-9df9f079acbb-kube-api-access-g9hxr" (OuterVolumeSpecName: "kube-api-access-g9hxr") pod "635134b8-0556-4ded-81d4-9df9f079acbb" (UID: "635134b8-0556-4ded-81d4-9df9f079acbb"). InnerVolumeSpecName "kube-api-access-g9hxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:55:30.554623 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.554599 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:30.554927 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.554915 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f8eb64-c9fb-4d30-9f12-1d03795f4251" containerName="main" Jul 15 13:55:30.554975 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.554928 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f8eb64-c9fb-4d30-9f12-1d03795f4251" containerName="main" Jul 15 13:55:30.554975 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.554948 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635134b8-0556-4ded-81d4-9df9f079acbb" containerName="test-curl-7422-140115524525632" Jul 15 13:55:30.554975 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.554955 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="635134b8-0556-4ded-81d4-9df9f079acbb" containerName="test-curl-7422-140115524525632" Jul 15 13:55:30.555065 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.555006 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f8eb64-c9fb-4d30-9f12-1d03795f4251" containerName="main" Jul 15 13:55:30.555065 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.555012 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="635134b8-0556-4ded-81d4-9df9f079acbb" containerName="test-curl-7422-140115524525632" Jul 15 13:55:30.557827 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.557812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:30.562674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.562652 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:30.628104 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.628078 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9hxr\" (UniqueName: \"kubernetes.io/projected/635134b8-0556-4ded-81d4-9df9f079acbb-kube-api-access-g9hxr\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:30.729361 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.729336 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvl9b\" (UniqueName: \"kubernetes.io/projected/2e3eab66-00bd-4de5-afea-9c9010cf6ac2-kube-api-access-mvl9b\") pod \"test-curl-7422-140115524525632\" (UID: \"2e3eab66-00bd-4de5-afea-9c9010cf6ac2\") " pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:30.830000 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.829932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvl9b\" (UniqueName: \"kubernetes.io/projected/2e3eab66-00bd-4de5-afea-9c9010cf6ac2-kube-api-access-mvl9b\") pod \"test-curl-7422-140115524525632\" (UID: \"2e3eab66-00bd-4de5-afea-9c9010cf6ac2\") " pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:30.837568 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.837533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvl9b\" (UniqueName: \"kubernetes.io/projected/2e3eab66-00bd-4de5-afea-9c9010cf6ac2-kube-api-access-mvl9b\") pod \"test-curl-7422-140115524525632\" (UID: \"2e3eab66-00bd-4de5-afea-9c9010cf6ac2\") " pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:30.866581 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.866562 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:30.992436 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:30.992402 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:30.994816 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:55:30.994791 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e3eab66_00bd_4de5_afea_9c9010cf6ac2.slice/crio-72645f42c76748fb3484aed17ee14bae2f00af085a83aa7919f1188bef41987b WatchSource:0}: Error finding container 72645f42c76748fb3484aed17ee14bae2f00af085a83aa7919f1188bef41987b: Status 404 returned error can't find the container with id 72645f42c76748fb3484aed17ee14bae2f00af085a83aa7919f1188bef41987b Jul 15 13:55:31.259311 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:31.259287 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:31.259459 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:31.259310 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfd50f558b12b26de64e2d75c924513095c2a96a5b08813c1946468046586705" Jul 15 13:55:31.260437 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:31.260414 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524525632" event={"ID":"2e3eab66-00bd-4de5-afea-9c9010cf6ac2","Type":"ContainerStarted","Data":"72645f42c76748fb3484aed17ee14bae2f00af085a83aa7919f1188bef41987b"} Jul 15 13:55:31.270029 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:31.270005 2582 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7422-140115524525632" oldPodUID="635134b8-0556-4ded-81d4-9df9f079acbb" podUID="2e3eab66-00bd-4de5-afea-9c9010cf6ac2" Jul 15 13:55:31.462278 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:31.462251 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635134b8-0556-4ded-81d4-9df9f079acbb" path="/var/lib/kubelet/pods/635134b8-0556-4ded-81d4-9df9f079acbb/volumes" Jul 15 13:55:32.265879 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:32.265845 2582 generic.go:356] "Generic (PLEG): container finished" podID="2e3eab66-00bd-4de5-afea-9c9010cf6ac2" containerID="e2de1e33b0dba1629b86f92672b46bd1114b5ae34477792c927e9cca05e9b12b" exitCode=0 Jul 15 13:55:32.266051 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:32.265930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140115524525632" event={"ID":"2e3eab66-00bd-4de5-afea-9c9010cf6ac2","Type":"ContainerDied","Data":"e2de1e33b0dba1629b86f92672b46bd1114b5ae34477792c927e9cca05e9b12b"} Jul 15 13:55:33.407774 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:33.407748 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:33.428233 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:33.428198 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7422-140115524525632_2e3eab66-00bd-4de5-afea-9c9010cf6ac2/test-curl-7422-140115524525632/0.log" Jul 15 13:55:33.452735 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:33.452684 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:33.454175 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:33.454148 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-140115524525632"] Jul 15 13:55:33.549479 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:33.549399 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvl9b\" (UniqueName: \"kubernetes.io/projected/2e3eab66-00bd-4de5-afea-9c9010cf6ac2-kube-api-access-mvl9b\") pod \"2e3eab66-00bd-4de5-afea-9c9010cf6ac2\" (UID: \"2e3eab66-00bd-4de5-afea-9c9010cf6ac2\") " Jul 15 13:55:33.551434 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:33.551411 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3eab66-00bd-4de5-afea-9c9010cf6ac2-kube-api-access-mvl9b" (OuterVolumeSpecName: "kube-api-access-mvl9b") pod "2e3eab66-00bd-4de5-afea-9c9010cf6ac2" (UID: "2e3eab66-00bd-4de5-afea-9c9010cf6ac2"). InnerVolumeSpecName "kube-api-access-mvl9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:55:33.650751 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:33.650704 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvl9b\" (UniqueName: \"kubernetes.io/projected/2e3eab66-00bd-4de5-afea-9c9010cf6ac2-kube-api-access-mvl9b\") on node \"ip-10-0-131-140.ec2.internal\" DevicePath \"\"" Jul 15 13:55:34.203041 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:34.203011 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:34.278762 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:34.278692 2582 scope.go:117] "RemoveContainer" containerID="e2de1e33b0dba1629b86f92672b46bd1114b5ae34477792c927e9cca05e9b12b" Jul 15 13:55:34.278925 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:34.278744 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140115524525632" Jul 15 13:55:34.303614 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:34.303588 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:35.460663 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:35.460628 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3eab66-00bd-4de5-afea-9c9010cf6ac2" path="/var/lib/kubelet/pods/2e3eab66-00bd-4de5-afea-9c9010cf6ac2/volumes" Jul 15 13:55:44.264087 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:44.264054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:47.687378 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:47.687279 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:55:48.741204 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:48.741170 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:54.812474 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:54.812429 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:54.915234 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:54.915203 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:59.440892 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:59.440858 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:55:59.703930 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:55:59.703898 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:00.344554 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:00.344512 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:00.443912 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:00.443879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:10.395174 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:10.395140 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:13.886491 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:13.886456 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8rqs9"] Jul 15 13:56:14.925432 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:14.925398 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:20.871703 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:20.871670 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:20.972865 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:20.972831 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:25.479553 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:25.479524 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:56:25.485778 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:25.485756 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zwvqz_0ec07d3f-257e-471b-8401-5b2ab083db2d/ovn-acl-logging/0.log" Jul 15 13:56:25.671325 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:25.671286 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:25.927672 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:25.927643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:31.999984 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:31.999957 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9bswr_786c6d59-4d31-48a3-a4e7-0c73d768d911/manager/0.log" Jul 15 13:56:32.121047 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:32.121024 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7lz7x_216f8da5-74c0-425a-ba2c-e49228fc7295/manager/0.log" Jul 15 13:56:32.383620 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:32.383558 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rp7bt_285afc68-cca3-4543-b1f7-dd5e0d7e1f2e/server/0.log" Jul 15 13:56:32.504777 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:32.504753 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vs5qs_d5ec325f-c272-40f8-a9dc-2b55642f757d/manager/0.log" Jul 15 13:56:32.621478 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:32.621455 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869c48857d-5w6tq_e339bef8-5ebf-4aad-b81c-a54ca8c1d23d/copy-manifests/0.log" Jul 15 13:56:32.629276 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:32.629256 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869c48857d-5w6tq_e339bef8-5ebf-4aad-b81c-a54ca8c1d23d/manager/0.log" Jul 15 13:56:36.549099 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:36.549068 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nth5w_7c776e71-9420-44f4-a077-1969278300a1/manager/0.log" Jul 15 13:56:36.668617 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:36.668588 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-svgjg_0cb6207b-86f6-4769-a4e7-c274b6801a68/kuadrant-console-plugin/0.log" Jul 15 13:56:36.791220 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:36.791191 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-44zf4_6d5f6fee-606e-4d74-855e-c55b0ec245ce/registry-server/0.log" Jul 15 13:56:36.922482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:36.922410 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-87l8m_3a03ef6b-6857-4629-9bd7-fd0ba3a94931/manager/0.log" Jul 15 13:56:37.065181 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:37.065154 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8rqs9_6086fdef-2573-413b-98c9-2f0b0e06d53e/limitador/0.log" Jul 15 13:56:37.184211 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:37.184146 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tzzjr_e567bcad-3f98-45b3-90c7-33466e6c83c0/manager/0.log" Jul 15 13:56:37.545589 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:37.545561 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678v6d2s_bc762472-b253-4d72-a6c1-6635a0b10a39/istio-proxy/0.log" Jul 15 13:56:37.784635 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:37.784609 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b6cfc997-56nkd_9051a7d5-2c0f-46d8-834f-8413e7996134/kube-auth-proxy/0.log" Jul 15 13:56:38.024590 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:38.024565 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2mlzm_5189b6ce-d057-47a4-80bb-6f89a2eb4b69/istio-proxy/1.log" Jul 15 13:56:39.191160 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:39.191133 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2g7fc_83874060-a0a8-4749-851f-dcd8262ce70d/main/0.log" Jul 15 13:56:39.304679 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:39.304651 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-cwzn5_eb61a06f-a9d4-4f75-a4c8-64d5b208578f/main/0.log" Jul 15 13:56:45.156606 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:45.156578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:45.253301 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:45.253269 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:45.352942 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:45.352917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" limit=1024 Jul 15 13:56:47.860534 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:47.860504 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6hjrv_62b177c1-f250-4a0e-9cbb-609fc3ffc741/global-pull-secret-syncer/0.log" Jul 15 13:56:47.990820 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:47.990794 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9jcjm_d709da95-0380-4aed-ad23-c7d7598e037a/konnectivity-agent/0.log" Jul 15 13:56:48.078307 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:48.078279 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-140.ec2.internal_6517c9ee24f3128cfced7da2acf57315/haproxy/0.log" Jul 15 13:56:52.837645 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:52.837618 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nth5w_7c776e71-9420-44f4-a077-1969278300a1/manager/0.log" Jul 15 13:56:52.868445 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:52.868420 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-svgjg_0cb6207b-86f6-4769-a4e7-c274b6801a68/kuadrant-console-plugin/0.log" Jul 15 13:56:52.909663 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:52.909639 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-44zf4_6d5f6fee-606e-4d74-855e-c55b0ec245ce/registry-server/0.log" Jul 15 13:56:53.057205 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:53.057175 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-87l8m_3a03ef6b-6857-4629-9bd7-fd0ba3a94931/manager/0.log" Jul 15 13:56:53.103683 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:53.103612 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8rqs9_6086fdef-2573-413b-98c9-2f0b0e06d53e/limitador/0.log" Jul 15 13:56:53.167143 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:53.167121 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tzzjr_e567bcad-3f98-45b3-90c7-33466e6c83c0/manager/0.log" Jul 15 13:56:55.295105 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:55.295075 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xskc4_bb13fb50-fa90-4093-b62f-d7d89ab1db54/node-exporter/0.log" Jul 15 13:56:55.318253 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:55.318223 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xskc4_bb13fb50-fa90-4093-b62f-d7d89ab1db54/kube-rbac-proxy/0.log" Jul 15 13:56:55.342815 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:55.342793 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xskc4_bb13fb50-fa90-4093-b62f-d7d89ab1db54/init-textfile/0.log" Jul 15 13:56:58.293034 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.292997 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn"] Jul 15 13:56:58.293414 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.293317 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3eab66-00bd-4de5-afea-9c9010cf6ac2" containerName="test-curl-7422-140115524525632" Jul 15 13:56:58.293414 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.293330 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3eab66-00bd-4de5-afea-9c9010cf6ac2" containerName="test-curl-7422-140115524525632" Jul 15 13:56:58.293414 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.293415 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e3eab66-00bd-4de5-afea-9c9010cf6ac2" containerName="test-curl-7422-140115524525632" Jul 15 13:56:58.296585 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.296565 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.298594 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.298575 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mvq2f\"/\"openshift-service-ca.crt\"" Jul 15 13:56:58.298669 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.298581 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mvq2f\"/\"kube-root-ca.crt\"" Jul 15 13:56:58.298669 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.298633 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mvq2f\"/\"default-dockercfg-wct6h\"" Jul 15 13:56:58.305541 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.305520 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn"] Jul 15 13:56:58.383896 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.383864 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-lib-modules\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.384059 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.383920 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-sys\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.384059 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.383949 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-proc\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.384059 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.383972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-podres\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.384059 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.383991 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnscc\" (UniqueName: \"kubernetes.io/projected/cdb667f6-d848-43fa-a28e-e3b46adc5e19-kube-api-access-hnscc\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.484939 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.484905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-sys\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.484939 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.484944 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-proc\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.485138 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.484962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-podres\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.485138 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.484983 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnscc\" (UniqueName: \"kubernetes.io/projected/cdb667f6-d848-43fa-a28e-e3b46adc5e19-kube-api-access-hnscc\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.485138 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.485022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-lib-modules\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.485138 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.485039 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-sys\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.485138 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.485043 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-proc\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.485138 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.485096 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-podres\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.485331 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.485146 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdb667f6-d848-43fa-a28e-e3b46adc5e19-lib-modules\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.492591 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.492568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnscc\" (UniqueName: \"kubernetes.io/projected/cdb667f6-d848-43fa-a28e-e3b46adc5e19-kube-api-access-hnscc\") pod \"perf-node-gather-daemonset-pssxn\" (UID: \"cdb667f6-d848-43fa-a28e-e3b46adc5e19\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.606941 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.606856 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:58.734212 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:58.734188 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn"] Jul 15 13:56:58.736292 ip-10-0-131-140 kubenswrapper[2582]: W0715 13:56:58.736266 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcdb667f6_d848_43fa_a28e_e3b46adc5e19.slice/crio-865affb5bebab98617ae734788cdbf5579356886d5b54c8f76aa123c0b332c5f WatchSource:0}: Error finding container 865affb5bebab98617ae734788cdbf5579356886d5b54c8f76aa123c0b332c5f: Status 404 returned error can't find the container with id 865affb5bebab98617ae734788cdbf5579356886d5b54c8f76aa123c0b332c5f Jul 15 13:56:59.683101 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:59.683063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" event={"ID":"cdb667f6-d848-43fa-a28e-e3b46adc5e19","Type":"ContainerStarted","Data":"d0a345849b5d730b229bb9905ecc6b01d828c183c3daacc8d94d61625abe4825"} Jul 15 13:56:59.683494 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:59.683146 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" event={"ID":"cdb667f6-d848-43fa-a28e-e3b46adc5e19","Type":"ContainerStarted","Data":"865affb5bebab98617ae734788cdbf5579356886d5b54c8f76aa123c0b332c5f"} Jul 15 13:56:59.683494 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:59.683172 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:56:59.699097 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:56:59.699051 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" podStartSLOduration=1.699037674 podStartE2EDuration="1.699037674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:56:59.697202935 +0000 UTC m=+3034.867149306" watchObservedRunningTime="2026-07-15 13:56:59.699037674 +0000 UTC m=+3034.868984027" Jul 15 13:57:00.459538 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:00.459509 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mjlh5_e2e9cb69-d4ea-4457-9b22-29286c5cc772/dns/0.log" Jul 15 13:57:00.481007 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:00.480985 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mjlh5_e2e9cb69-d4ea-4457-9b22-29286c5cc772/kube-rbac-proxy/0.log" Jul 15 13:57:00.599955 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:00.599935 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjvdt_57661864-e119-4734-9e4b-f9932acf74e4/dns-node-resolver/0.log" Jul 15 13:57:01.193781 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:01.193753 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gptn8_4736cf14-4df8-4a5d-95c1-d026e0d88d42/node-ca/0.log" Jul 15 13:57:02.141804 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:02.141776 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678v6d2s_bc762472-b253-4d72-a6c1-6635a0b10a39/istio-proxy/0.log" Jul 15 13:57:02.279081 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:02.279055 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b6cfc997-56nkd_9051a7d5-2c0f-46d8-834f-8413e7996134/kube-auth-proxy/0.log" Jul 15 13:57:02.329828 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:02.329807 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2mlzm_5189b6ce-d057-47a4-80bb-6f89a2eb4b69/istio-proxy/0.log" Jul 15 13:57:02.359634 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:02.359609 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2mlzm_5189b6ce-d057-47a4-80bb-6f89a2eb4b69/istio-proxy/1.log" Jul 15 13:57:03.167866 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:03.167838 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghmn2_429fe006-ddc8-454f-b468-9a6d3f553856/serve-healthcheck-canary/0.log" Jul 15 13:57:03.832322 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:03.832298 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vxkpv_d6e424f9-072b-487b-b86d-50a57cd8bbc6/kube-rbac-proxy/0.log" Jul 15 13:57:03.852006 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:03.851978 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vxkpv_d6e424f9-072b-487b-b86d-50a57cd8bbc6/exporter/0.log" Jul 15 13:57:03.871857 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:03.871830 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vxkpv_d6e424f9-072b-487b-b86d-50a57cd8bbc6/extractor/0.log" Jul 15 13:57:05.702129 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:05.702102 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-pssxn" Jul 15 13:57:05.978685 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:05.978583 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9bswr_786c6d59-4d31-48a3-a4e7-0c73d768d911/manager/0.log" Jul 15 13:57:06.103952 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:06.103926 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7lz7x_216f8da5-74c0-425a-ba2c-e49228fc7295/manager/0.log" Jul 15 13:57:06.228917 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:06.228852 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rp7bt_285afc68-cca3-4543-b1f7-dd5e0d7e1f2e/server/0.log" Jul 15 13:57:06.295380 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:06.295354 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vs5qs_d5ec325f-c272-40f8-a9dc-2b55642f757d/manager/0.log" Jul 15 13:57:06.319107 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:06.319075 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869c48857d-5w6tq_e339bef8-5ebf-4aad-b81c-a54ca8c1d23d/manager/0.log" Jul 15 13:57:06.338950 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:06.338933 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869c48857d-5w6tq_e339bef8-5ebf-4aad-b81c-a54ca8c1d23d/copy-manifests/0.log" Jul 15 13:57:07.965688 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:07.965664 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8j9cx_ca5e3b86-2247-4e9b-8515-2926c8aaf0de/openshift-lws-operator/0.log" Jul 15 13:57:08.469438 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:08.469367 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678v6d2s_bc762472-b253-4d72-a6c1-6635a0b10a39/istio-proxy/0.log" Jul 15 13:57:08.584503 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:08.584456 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b6cfc997-56nkd_9051a7d5-2c0f-46d8-834f-8413e7996134/kube-auth-proxy/0.log" Jul 15 13:57:08.617931 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:08.617901 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2mlzm_5189b6ce-d057-47a4-80bb-6f89a2eb4b69/istio-proxy/0.log" Jul 15 13:57:08.647029 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:08.647009 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2mlzm_5189b6ce-d057-47a4-80bb-6f89a2eb4b69/istio-proxy/1.log" Jul 15 13:57:09.869475 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:09.869453 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678v6d2s_bc762472-b253-4d72-a6c1-6635a0b10a39/istio-proxy/0.log" Jul 15 13:57:09.990544 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:09.990463 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b6cfc997-56nkd_9051a7d5-2c0f-46d8-834f-8413e7996134/kube-auth-proxy/0.log" Jul 15 13:57:10.026674 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:10.026630 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2mlzm_5189b6ce-d057-47a4-80bb-6f89a2eb4b69/istio-proxy/0.log" Jul 15 13:57:10.056931 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:10.056886 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-2mlzm_5189b6ce-d057-47a4-80bb-6f89a2eb4b69/istio-proxy/1.log" Jul 15 13:57:14.582979 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:14.582952 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfl8z_1e2749e6-96d8-4e06-8bdc-0693774149e9/kube-multus-additional-cni-plugins/0.log" Jul 15 13:57:14.602482 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:14.602458 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfl8z_1e2749e6-96d8-4e06-8bdc-0693774149e9/egress-router-binary-copy/0.log" Jul 15 13:57:14.622348 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:14.622331 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfl8z_1e2749e6-96d8-4e06-8bdc-0693774149e9/cni-plugins/0.log" Jul 15 13:57:14.643350 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:14.643328 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfl8z_1e2749e6-96d8-4e06-8bdc-0693774149e9/bond-cni-plugin/0.log" Jul 15 13:57:14.662480 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:14.662460 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfl8z_1e2749e6-96d8-4e06-8bdc-0693774149e9/routeoverride-cni/0.log" Jul 15 13:57:14.681703 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:14.681683 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfl8z_1e2749e6-96d8-4e06-8bdc-0693774149e9/whereabouts-cni-bincopy/0.log" Jul 15 13:57:14.700249 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:14.700230 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfl8z_1e2749e6-96d8-4e06-8bdc-0693774149e9/whereabouts-cni/0.log" Jul 15 13:57:14.906969 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:14.906908 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n4cml_ec946d7b-a0ae-471b-90b1-b1735c2ade21/kube-multus/0.log" Jul 15 13:57:15.046902 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:15.046878 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cx6mz_b811d4da-54d9-42e2-b09c-ada3620a5a92/network-metrics-daemon/0.log" Jul 15 13:57:15.067356 ip-10-0-131-140 kubenswrapper[2582]: I0715 13:57:15.067330 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cx6mz_b811d4da-54d9-42e2-b09c-ada3620a5a92/kube-rbac-proxy/0.log"